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1. The Investing in Innovation (i3) Fund 


The Investing in Innovation (i3) Fund was established by the American Recovery and Reinvestment Act 
of 2009, which called on the U.S. Department of Education (ED) to provide student achievement awards 
to local education agencies and nonprofits with a record of improving student achievement.’ ED set 
priorities for the program and specified them in the program regulations and the invitation for 
applications.”* The program awards competitive grants to implement and evaluate educational 
interventions. 


The i3 program aligns the amount of funding awarded to the prior evidence on and the expected 
implementation scale of the proposed intervention. The smallest, or “Development,” grants support 
developing and testing interventions with limited or no prior evidence. Interventions with moderate 
evidence of effectiveness can receive a larger “Validation” grant, to implement and test the intervention in 
a broader population or in new contexts. The largest, or “Scale-up,” grants support interventions with 
strong prior evidence of effectiveness to be implemented and tested on a much larger scale. The 
differential funding and expectations for the three types of grants means that, in effect, i3 operates like 
three grant programs in one.* 


ED’s Office of Innovation and Improvement (OI) administers the i3 program. The i3 program requires 
grantees to conduct an independent evaluation to build evidence and identify effective educational 
interventions. OI] recognized that credible evidence is needed to identify effective interventions but is 
challenging to produce. To address this challenge, OII asked the Institute of Education Sciences (IES) to 
provide support for planning and conducting strong evaluations. 


IES contracted with Abt Associates Inc. and its partners (the Abt Team) to conduct the i3 Technical 
Assistance and Evaluation Project. The Abt Team’s two key activities were (1) providing technical 
assistance to support the design and conduct of strong evaluations, and (2) assessing the strength of and 
summarizing the findings from the i3 evaluations. This project built on prior work supporting grants with 
a requirement to conduct rigorous evaluations.” 


American Recovery and Reinvestment Act of 2009, Section 14007, Title XIV (Public Law 111- 
5). http://www2.ed.gov/policy/gen/leg/recovery/statutory/stabilization-fund.pdf 


U.S. Department of Education. (2010, March 12). Office of Innovation and Improvement; Investing in 
Innovation Fund; Final Rule and Notice. Federal Register, 75(48): 12004-12071. 
https://www.gpo.gov/fdsys/pkg/FR-2010-03-12/pdf/2010-5147.pdf 


U.S. Department of Education. (2010, March 12). Office of Innovation and Improvement; Overview 
Information: Investing in Innovation Fund; Notice Inviting Applications for New Awards for Fiscal Year (FY) 
2010. Federal Register, 75(48): 12072-12086. https://www.gpo.govw/fdsys/pkg/FR-2010-03-12/pdf/2010- 
5139.pdf 

In competitions after Fiscal Year 2010, the i3 program issued multiple Notices Inviting Applications for the 
three tiers, which is consistent with the characterization of the program as three programs in one. 


Boulay, B., Goodson, B., Frye, M., Blocklin, M., and Price, C. (2015). Summary of research generated by 
Striving Readers on the effectiveness of interventions for struggling adolescent readers (NCEE 2016-4001). 
Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education 
Sciences, U.S. Department of Education. http://ies.ed.gov/ncee/pubs/20164001/ 
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This first report of the i3 Technical Assistance and Evaluation Project summarizes the strength of and 
findings from the 67 i3 evaluations that were completed by May 2017. In addition, this report summarizes 
the extent to which the i3 program met its goal of contributing to the evidence base on effective 
educational interventions. Below, we describe the i3 program in more detail. 


1.1 The i3 Program Logic Model 


The Abt Team developed a logic model to describe the program and provide an organizing framework for 
this report. The i3 program logic model in Exhibit 1.1, therefore, is our interpretation of the key aspects of 
the i3 program. We describe the i3 program as based on a theory of action that includes a set of priorities, 
which lead to the implementation of the key program activities, in order to meet the intended short- and 
long-term goals of the program. 


1.1.1 i3 Program Priorities 


We identified three i3 program priorities from the program regulations: (1) fund tiers of 
grants, (2) require independent evaluations, and (3) fund a broad portfolio of educational 
interventions. We describe the implementation of the i3 program priorities in more detail 
in Chapter 2. 


Fund tiers of grants. The i3 program awards its largest grants to the interventions with the strongest prior 
evidence of effectiveness and smaller grants to programs with little or no prior evidence. To date, the i3 
program has awarded more than $1.4 billion to 172 grantees in three tiers: 


e $340 million (24 percent) to 11 Scale-up grants, 
e $702 million (49 percent) to 46 Validation grants, and 


e $386 million (27 percent) to 115 Development grants. 


Require independent evaluations. The i3 program requires that each grantee fund an independent 
evaluation that estimates the impact of the i3-funded intervention as implemented at the proposed level of 
scale. ED provided additional details about expectations for these evaluations in the program’s 
performance measures and selection criteria.° The grantees were also required to cooperate with ED’s 
technical assistance and to make the results of their evaluation publicly available. 


Fund a broad portfolio of interventions. Unlike many ED grant programs that focus on a specific content 
area, such as reading, i3 funds a broad portfolio of educational interventions that serve high-need students 
in kindergarten through grade 12. Each year, the i3 program selects absolute priorities. In FY 2010, for 
example, the absolute priorities were to fund innovations that (1) supported effective teachers and 
principals, (2) improved the use of data, (3) complemented the implementation of high standards and 
high-quality assessments, and (4) turned around persistently low-performing schools. Selecting absolute 
priorities allows the i3 program to focus within the overall broad scope of education issues for high-need 
elementary and secondary students. 


° U.S. Department of Education (2010), pp. 12080-12086. 
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1.1.2 Key Program Activities 


Three of the six key program activities are related to the required evaluations and are 
shown in the white boxes in Exhibit 1.1.’ The Abt Team conducted these three 
evaluation-related program activities under contract to IES, as summarized below and in 
more detail in Chapter 3. 


Adopt or develop criteria to assess the strength of the i3 evaluations. A short-term goal of the i3 
program is to produce strong evaluations. Assessments of whether the evaluations meet the short-term 
goal of i3 rest on establishing clear criteria against which to assess the evaluations. The Abt Team 
adopted existing criteria and developed additional criteria to assess the strength of the i3 evaluations. 
These criteria formed the basis for (1) the Abt Team’s periodic review and assessment of the design and 
conduct of the i3 evaluations, and (2) the provision of technical assistance to evaluators. 


Support design and conduct of evaluations. The Abt Team provided comprehensive technical assistance 
to ensure that the i3 evaluators used the strongest possible evaluation designs to determine the impacts of 
the i3 interventions and provide high-quality implementation data. The Abt Team provided ongoing 
technical assistance to help evaluators identify and mitigate risks to the strength of their evaluations. 
Finally, Abt Team members, separate from those delivering technical assistance, periodically reviewed 
the strength of the evaluations. Abt Team reviewers assessed whether the proposed evaluation designs 
would meet the criteria for high-quality implementation evaluations and strong impact evaluations. Their 
assessments informed the technical assistance and were communicated to both the evaluators and ED. 


Collect data on the conduct of and findings from the i3 evaluations. The Abt Team conducted a 
systematic data collection of information needed to (1) summarize the findings from the completed 
evaluations, and (2) put the findings in the context of the strength of the evaluations that produced them. 
Evaluators participated in the data collection near the end of the grant period. We focused our data 
collection efforts on findings that were the result of analyses that the evaluators had prespecified; that is, 
analyses they had committed in advance to conducting and reporting to the Abt Team. The prespecified 
analyses were the focus of both our technical assistance and our periodic reviews of the strength of the 
evaluations, and they are the focus of this report’s summary of findings. 


The three key program activities shown in the shaded boxes in the middle panel of Exhibit 1.1 were conducted 
by i3 program staff or their contractors: provide grant management, provide support for creation of communities 
of practice, and provide technical assistance on the delivery of the interventions. Summarizing these activities is 
beyond the scope of this report. 
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Exhibit 1.1: 


Priorities 


Fund tiers of grants that 
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Fund a broad portfolio of 
interventions to improve 
academic outcomes for 
high-need students 


* Establish absolute 
priorities annually that 
define areas for funding 


Academic achievement awards to local education agencies and nonprofits 
with a record of improving student achievement 


i3 Program 


i3 Program Logic Model 


Provide grant management 


Provide support for creation of 
communities of practice among i3 grantees 


Provide TA to grantees on intervention 
delivery 


Adopt or develop criteria to assess the 
strength of the i3 evaluations 


Support design and conduct of evaluations 


* Conduct initial needs assessment 

* Provide TA to evaluators to strengthen 
evaluation designs and conduct rigorous 
evaluations 

* Identify and resolve risks to the 
strength of evaluations (continuous risk 
assessment) 

* Periodically assess the strength of the 
evaluations and provide feedback to 
evaluators and ED 


Collect data on the conduct of and findings 
from the i3 evaluations 


Program Goals 


—. 


Short-Term 


High-quality 
implementation 
of the i3-funded 
interventions 


Strong evaluations: 


Implementation 
evaluations provide 
high-quality data 


on implementation 
fidelity 


Impact evaluations 
are independent, 
meet What Works 
Clearinghouse 
evidence 
standards, and 
adequately 
represent the 
population served 


Long-Term 


Effective practices 
reach larger 
numbers of high- 
need students 


Build evidence 
for effective 
interventions at 
increasing scale: 


* Implementation 
evaluations 
find that the 
interventions were 
implemented with 
adequate fidelity 


Impact evaluations 
find that the 
interventions had at 
least one positive 
impact on a student 
academic outcome 


Note: The Key Program Activities in the shaded boxes were provided by Oll and their contractors. Those activities are expected to lead to the 
Program Goals in the shaded boxes. Describing the activities and goals in the shaded boxes is beyond the scope of this report. 
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1.1.3 i3 Program Goals 


The i3 program aims to build the evidence base on effective educational 
6 interventions.® The short-term goal of the i3 program is therefore strong 
evaluations that provide high-quality data on implementation fidelity and 


~~, 


EVIDENCE 


C 


“aa independent impact evaluations that meet What Works Clearinghouse™ 
(WWC) Standards” and adequately represent the population served by the 
intervention. The long-term goal of i3 is to identify effective interventions that can be implemented with 
adequate fidelity at increasing scale. 


1.2 Roadmap to this Report 


The remainder of this report describes the implementation of the i3 program priorities and key program 
activities and summarizes the findings from the 67 i3 evaluations completed by May 2017 to assess the 
extent to which the i3 program achieved its goals. 


Chapter 2 describes the implementation of the i3 program priorities, and describes the 
interventions that ED funded grantees to implement and evaluate. 


Chapter 3 describes the implementation of the three i3 key program activities that were related to 
the i3 evaluations. 


Chapter 4 describes the extent to which the i3 program achieved its short-term goal by 

ewoexcel describing the strength of the i3 evaluations that have been completed by May 2017. It also 
describes the extent to which the i3 program met its long-term goal of identifying effective 
interventions that can be implemented with adequate fidelity (as defined by the grantees and 
evaluators). 


OOo 


Chapter 5 discusses the extent to which the four Scale-up grantees met their self-identified goals for 


scaling their interventions. 


Finally, Chapter 6 offers lessons learned for other grant programs considering a rigorous evaluation 
requirement. 


loc} 


The program goals shown in the shaded boxes in the right-hand panel of Exhibit 1.1 are the expected outcomes 
of the key program activities delivered by the i3 program staff and their contractors; summarizing these goals is 
beyond the scope of this report. 


It is important to note that only reviews conducted under a WWC contract can produce an official WWC rating 
for a study. Thus, the evidence assessments we conducted for this report are not a substitute for a WWC review, 
and we can only give an evaluation an unofficial WWC rating (see Section 4.1.3 for more details). 
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2. Implementation of the i3 Program Priorities 


The i3 program can be characterized as having three priorities (see Exhibit 1.1): (1) fund 
tiers of grants, (2) require independent evaluations of the i3-funded interventions, and (3) 
fund a broad portfolio of interventions that aim to improve academic outcomes for high- 
need students. In this chapter, we describe the implementation of the three i3 program 
priorities in the sample of 67 i3 grants completed by May 2017. The i3 program funded 
grants through Fiscal Year (FY) 2016, some of which will not end until December 2021. While this 
sample of 67 grants might not be representative of all i3 grants, the findings in this report are informative 
about the i3 program in its early years.'” 


2.1 Fund Tiers of Grants 


The 67 grants included in this report were awarded a total of $679 million, or 49 percent of the total i3 
funds awarded to date. This includes: 


e $195 million (29 percent of the $679 million) to four Scale-up grants, which could receive up to 
$50 million each. These constitute four of the 11 Scale-up grants (36 percent) awarded by the i3 
program to date. 


e $292 million (43 percent) to 15 Validation grants, which could receive up to $25 million each in 
FY 2010 and up to $15 million each in FY 2011 and FY 2012. These constitute 15 of the 46 
Validation grants (33 percent) awarded by the i3 program to date. 


e $192 million (28 percent) to 48 Development grants, which could receive up to $5 million each 
in FY 2010 and up to $3 million each in FY 2011 and FY 2012. These constitute 48 of the 115 
Development grants (42 percent) awarded by the i3 program to date. 


By design, the i3 program awarded a large share of the funds (almost one-third) to a small number of 
Scale-up grantees that proposed to implement interventions supported by strong prior evidence on a large 
scale. Many more Development grants were awarded a similar share of the funds to further develop and 
test the effectiveness of interventions with little or no prior evidence of effectiveness. 


2.2 Require Independent Evaluations 


The i3 program requires that each grantee propose and conduct an independent evaluation of the i3- 
funded intervention. To meet this requirement, the grantees named a separate evaluation team in their 
applications. The grantees tended to hire external organizations to conduct their evaluations, thereby 
laying the groundwork for independence between those implementing the intervention and those 
evaluating its effectiveness. The Abt Team’s review of the grant applications determined that for all 67 
grants, the proposed evaluator was affiliated with a different organization than the organization that 
would be responsible for developing and implementing the intervention. 


10 ED plans to release updated versions of the tables and figures included in this report to summarize the strength 


of and findings from i3 evaluations completed after May 2017. 
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The evaluators hired by the 67 grantees in this report include a range of organizations, from individual 
consultants to large national research firms. Among the grantees, 24 chose national research firms as their 
evaluation partner, 22 chose smaller research firms or individual consultants, and 21 grantees chose to 
work with a university partner. 


Of the $679 million awarded to these grantees, $99 million (15 percent) was spent on evaluation. On 
average, grantees spent 16 percent of their grant award on evaluation, with the evaluation budgets ranging 
from 3 to 38 percent of the grants. On average, Scale-up grantees spent approximately $4 million on 
evaluation (8 percent of the grant award), Validation grantees spent approximately $3.5 million on 
evaluation (17 percent of the grant award), and Development grantees spent approximately $645,000 on 
evaluation (16 percent of the grant award). 


The grantees are required to make the results of their evaluations publicly available. As of March 2018, 
47 of all 172 grants funded (27 percent) have produced public reports. Of the 67 completed evaluations in 
this report, 46 (69 percent) have produced public reports. '' 


2.3 Fund a Broad Portfolio of Interventions 


The i3 program established a set of absolute priorities that describe educational barriers to the 
achievement of high-need students. The 67 completed grants included in this report identified in their 
applications the priority that their proposed intervention addresses. Because the grants in this report span 
the FY 2010, FY 2011, and FY 2012 funding years, the absolute priorities that the grantees identified 
include absolute priorities from each of the three years (Exhibit 2.1). 


Exhibit 2.1: i3 Program Absolute Priorities Identified by the 67 Completed Grants 


Scale-up | Validation | Development 


Grants Grants Grants 
Absolute Priority Grant Year (n=4) (n=15) (n=48) 
Innovations that Improve the Effectiveness and FY 2010, 9 3 10 15 
Distribution of Effective Teachers or Principals FY 2011 
Innovations that Turn Around Persistently Low- FY 2010, 
Performing Schools FY 2011, 2 3 10 15 
FY 2012 
Innovations that Complement the Implementation of FY 2010, 0 5 14 19 
High Standards and High-Quality Assessments FY 2011 
Innovations that Improve the Use of Data FY 2010 0 2 6 8 
Innovations that Promote Science, Technology, 
Engineering, and Mathematics (STEM) Education pe : ; . : 
Improving Achievement and High School Graduation FY 2011, 0 1 9 3 
Rates in Rural Educational Agencies FY 2012 
Improving School Engagement, School Environment, 
and School Safety and Improving Family and FY 2012 0 0 2 2 
Community Engagement 


"The i3 program lists the available reports here: https://innovation.ed.gov/what-we-do/innovation/investing-in- 


innovation-i3/awards/ 
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Each grantee provided a narrative description of the intervention proposed in their applications. However, 
only 39 percent included a detailed logic model depicting the theory by which the intervention is expected 
to improve student academic outcomes. Only 15 percent described existing methods for assessing fidelity 
of implementation. As part of our technical assistance (see Section 3.2 below), the Abt Team worked with 
evaluators to develop an intervention logic model (or to revise and expand an initial logic model) that 
shows how the intervention’s key components (i.e., the primary activities conducted by those 
implementing the intervention) are expected to change mediators (i.e., behaviors, processes, and skills) 
through which the intervention was expected to ultimately have an impact on short-term student 
outcomes and longer-term student outcomes. The framework for the logic models was provided by the 
Abt Team and took the form shown in Exhibit 2.2. The number of boxes in each column reflects the most 
common number of each element across the 67 intervention logic models. * 


Exhibit 2.2: Generic i3 Intervention Logic Model 


Key Components Mediators Student Short-Term Student Longer-Term 
Outcomes Outcomes 


Key Component 1 Mediator 1 


Mediator 2 Short-Term Outcome 1 


Mediator 3 


Longer-Term Outcome 1 


Key Component 2 


Key Component 3 


Longer-Term Outcome 2 


Key Component 4 


(in 
it 


The logic models developed by the grantees and evaluators include a large number of different key 
components and mediators, as well as different short- and longer-term student outcomes. The logic 
models for the 67 interventions summarized in this report specify 329 key components, 414 mediators, 
151 short-term student outcomes, and 221 longer-term student outcomes. 


The Abt Team coded each element of the logic models to summarize the variety of approaches the 
interventions took to improve student academic achievement. The coding scheme included 11 categories 
of key components, 3 categories each of mediators and short-term student outcomes, and 4 categories of 
longer-term student outcomes. The following sections summarize the Abt Team’s coding of the 67 
intervention logic models. 


2.3.1 Key Components Implemented by the Intervention Developers 


The i3 program funded a broad portfolio of interventions, so it is not surprising that the logic models 
included a range of both individual key components and combinations of key components. Most 
interventions specified four key components in their logic models; all but one of the interventions had at 
least two key components and nine had eight or more.'* The number of key components ranged from 1 to 
24, Nine of the 67 interventions had 8 or more key components. 


12 Equal numbers of grants included three or four mediators (nine grants each). The generic logic model displays 


the lower number of elements. 


‘3 One Scale-up intervention, KIPP, characterized its intervention with a single key component. See Appendix A. 
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The most common category of key components was Provide Professional Development, defined as direct 
training to school staff to improve practice (Exhibit 2.3). Eighty-five percent of the interventions included 
at least one professional development activity as a key component of the intervention. Of those, 29 
interventions (51 percent) provided professional development solely to teachers, and 23 interventions (40 
percent) provided professional development to both teachers and other school staff. A smaller number of 
interventions (5 interventions, 9 percent) provided professional development solely to other school staff 
(e.g., counselors, administrators) or non-school staff (e.g., mentors, local professionals). 


Exhibit 2.3: Key Components Identified in the i3 Intervention Logic Models 


Provide Professional Development 85% 
Develop/Institute New Curriculum & Materials 
Support Staff Collaboration 

Provide Coaching 

Target Leadership Structures and Supports 
Select/Evaluate Staff 


Plan for and Support Assessment & Data Use 


Involve Parents/Community Members 


Institute Structural Changes 13% 
12% 


10% 
T 


Support College/Career Readiness 


Provide Services Targeting Individualized Learning 


T T 1 


0% 20% 40% 60% 80% 100% 


Coaching (Provide Coaching) was categorized separately, as another mode of professional development, 
and was included in 22 percent of interventions. The only other category of key component that was 
included by a majority of interventions was Developing and Instituting New Curriculum and Materials 
into classrooms (60 percent of interventions). 


There was a wide variety of specific program activities within each of the 11 categories of key 
components. For example, Institute Structural Changes includes activities such as introduction of flexible 
student scheduling, extending the school year, and establishing a student technology center in the school. 
Provide Services Targeting Individualized Learning includes activities such as individual student 
mentoring and individualized case management for students, especially for high-need students. (See 
Appendix A for descriptions of all 67 interventions and the key components named in their logic models.) 


2.3.2. Mediators through which Interventions are Expected to Improve Short- and Longer-Term 
Student Outcomes 


The logic models also describe the mediators that are the behaviors, processes, and skills targeted by each 
intervention’s key components. Mediators are expected, in turn, to lead to improved student outcomes. 
The Abt Team coded the 414 mediators into one of three categories: (1) changes in school structures or 
processes (School Mediators), (2) changes in classrooms (Classroom Mediators), or (3) changes in 
families (Family Mediators). 
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Examples of school mediators include: 


e development of schoolwide norms and processes for continuous improvement; 
e use of a facilitative leadership approach by the principal; 


e joint teacher and administrator analysis of formative assessments to support data-driven decision- 
making; 


e development of school action plans; and 


e increased learning time through extension of the school day or school year (e.g., summer instructional 
activities). 


Examples of classroom mediators include: 


e integration of technology into science instruction; 

e use of inquiry-based approaches for teaching content area skills; 

e increased individualization of classroom instruction and better classroom management; and 

e integration of the arts into reading and writing through a variety of art forms and literary genres. 


Examples of family mediators include: 


e increasing support for the child’s education at home; 

e increasing family involvement in school activities; and 

e developing family-school partnerships. 

Three-quarters (76 percent) of the intervention logic models specified mediators in more than one of the 


three categories (Exhibit 2.4). Slightly more than half of the intervention logic models included both 
school and classroom mediators (but not family mediators). 


Exhibit 2.4: Mediators Identified in the i3 Intervention Logic Models 


School and Classroom Mediators 52% 
School Mediators Only 

School, Classroom, and Family Mediators 
School and Family Mediators 

Classroom Mediators Only 


Family Mediators Only 


Classroom and Family Mediators 


0% 20% 40% 60% 80% 100% 


2.3.3. Short- and Longer-Term Student Outcomes 


While all of the i3 interventions aimed to improve student academic outcomes, most also specified short- 
term (or intermediate) outcomes on the path to longer-term achievement and attainment outcomes (54 
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interventions, or 81 percent). The Abt Team coded the 151 short-term student outcomes in the 67 logic 
models into three categories: 


e Approaches to Learning, including 21st-century skills, use of strategies/methods for learning and 
analysis, inquiry skills, and effective study habits; 


e Engagement in school, including greater interest in school, engagement in school activities, more 
time on task, and better school attendance; and 


e Interests and Beliefs, including higher college and career aspirations and greater confidence and sense 
of self-efficacy, and positive relations with other members of the school community. 


Approaches to Learning and Engagement were each identified as short-term student outcomes in 
approximately half of the 67 logic models (57 percent and 46 percent, respectively; Exhibit 2.5). 


Exhibit 2.5: Short-term Student Outcomes Identified in the i3 Intervention Logic Models 


Approaches to Learning 57% 
Engagement 46% 
Interests and Beliefs 33% 
0% 20% 40% 60% 80% 100% 


All of the i3-funded interventions were expected to aim to improve longer-term student achievement and 
attainment outcomes. The i3 intervention logic models included 221 longer-term outcomes that the Abt 
Team coded into one of the four domains of interest to the i3 program: (1) Improving Academic 
Achievement (including overall achievement or closing achievement gaps), (2) Decreasing Dropout 
Rates, (3) Increasing High School Graduation Rates (including indicators in earlier grades of being on 
track for graduating), and (4) Improving College and Career Outcomes (including college enrollment and 
readiness, completion rates, and labor market outcomes). As shown in Exhibit 2.6, all 67 interventions 
included Academic Achievement as a longer-term student outcome in their logic models. Around one- 
third of the interventions included Improving College and Career Outcomes, and one quarter included 
Increasing High School Graduation Rates. 


Exhibit 2.6: Longer-term Student Outcomes Identified in the i3 Intervention Logic 
Models 


Improving Academic Achievement 
Improving College and Career Outcomes 


Increasing High School Graduation Rates 


Decreasing Dropout Rates 


0% 20% 40% 60% 80% 100% 
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Within the Academic Achievement category, the 67 interventions included outcomes in the core academic 
subjects (English Language Arts, mathematics, science, and social studies), as well as in other subjects 
such as art (Exhibit 2.7). 


Exhibit 2.7: Types of Student Achievement Outcomes Identified in the i3 Intervention 
Logic Models 


English Language Arts 


Mathematics 


69% 


General Achievement 
Science 
Academic Performance in School 


Social Studies 


Achievement in Other Subjects 


T T T T 1 


0% 20% 40% 60% 80% 100% 
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3. Implementation of the Three Key Program Activities Related to 


Evaluation 


The federal investment in the i3 grants is substantial; consequently, ED aims to learn as 
much as possible from the i3 evaluations. ED expects the i3 grants to sponsor strong 
evaluations to build evidence on effective interventions. In addition to the funds awarded 
to grantees, ED invested approximately $30,000 per grant per year, or $10 million total, 
to support the comprehensive technical assistance provided to the 67 evaluations 
summarized in this report.'* This level of investment in direct support for evaluation is unusual among 
grant programs and reflects ED’s high expectations for what would be learned from the i3 program. It is 
also important to put the strength of the completed i3 evaluations (see Chapter 4) in the context of these 
substantial supports. 


The three key program activities related to the i3 evaluations are (1) adopt or develop criteria to assess the 
strength of the i3 evaluations, (2) support the design and conduct of the i3 evaluations, and (3) collect data 
on the conduct of and findings from the i3 evaluations. This chapter summarizes how the Abt Team 
conducted these three activities. 


3.1 Adopt or Develop Criteria to Assess the Strength of the i3 Evaluations 


The Abt Team adopted or developed criteria to assess the strength of i3 evaluations. These criteria guided 
technical assistance we provided the evaluators. They also were used to provide periodic assessments of 
the strength of the evaluations to the grantees and ED. More specifically, the Abt Team: 


e Adopted the WWC Evidence Standards (version 3.0) to assess the strength of the impact evaluations 
and developed a review protocol to ensure the consistent application of those standards to all i3 
impact evaluations (see Appendix B), including those that do not fall under existing WWC review 
protocols; ‘> '° 


'* This cost estimate does not include collecting evaluation data from the grantees, the third key activity described 


in this section. 


'S Tt is important to note that the reviews reported here differ from an official WWC review in two important 


ways. First, the data available to the Abt Team differ from the data available to the WWC. The Abt Team 
collected the information necessary for the reviews directly from the evaluators and limited the summary of 
impact findings to those that were prespecified. The WWC limits its reviews to publicly available reports of 
evaluation findings. Second, only reviews conducted under a WWC contract can produce an official WWC 
rating for a study. Thus, the evidence reviews we conducted for this report are not a substitute for a WWC 
review, and we can only give an evaluation an unofficial WWC rating. For these reasons, the official WWC 
rating of the i3 evaluations can differ from the ratings reported here. We include our unofficial ratings in this 
report because meeting these standards is a key goal of the i3 program and because it is important to place this 
summary of evaluation findings in the context of the strength of the evaluations. 


‘© The Abt Team also developed criteria to assess whether evaluations funded by Development grants provided 


evidence on the intervention’s promise. In the FY 2010-FY 2014 competitions, impact evaluations supported 
by Development grants were expected to provide evidence on the promise of the intervention for improving 
student outcomes. Though evidence of promise can support further development and testing of the intervention, 
it does not support conclusions that the intervention caused the observed effects on student outcomes, making 
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e Developed criteria for assessing the strength of implementation data and performance feedback, 
including whether an implementation evaluation provided information on key features of the program 
for future replication or testing in the form of a comprehensive logic model and periodically measured 
fidelity of implementation of the key components of the intervention model; 


e Developed criteria for whether an impact evaluation was conducted independent of those responsible 
for developing or implementing the i3 intervention; and 


e Developed criteria for whether an impact evaluation sample was representative of those served by 
each i3 intervention. 


The Abt Team developed definitions for the strength of an implementation evaluation, independence, and 
representativeness. The definitions were operationalized into criteria that were communicated to grantees 
and evaluators and that guided our delivery of TA aimed at strengthening the evaluations. The criteria 
represent the Abt Team’s best judgment in developing a framework for assessing the evaluations along 
these dimensions. 


The task of developing criteria for implementation evaluations was especially challenging. The field of 
implementation science has generated a range of approaches and best practices for measuring and 
summarizing the implementation of an intervention. '’ However, we are unaware of any attempts to codify 
best practices into a set of criteria that assesses whether a particular evaluation was well conducted. As 
indicated earlier, most of the grantees had blueprints for implementing their interventions but had not 
developed well-specified logic models or methods of measuring fidelity of implementation.'® The Abt 
Team developed criteria that allowed grantees substantial latitude to specify what the key components of 
their interventions were, how the implementation of these components should be measured, and what 
qualified as adequate fidelity of each component. The Abt Team did not include measurement of 
implementation in the comparison group in the criteria because this was not one of ED’s expectations, 
although we encouraged them to do so as good practice. 


the evidence less useful to those considering adopting an intervention. In the FY 2015 competition and beyond, 
the selection criteria for Development grant awards included proposing an evaluation that would meet WWC 
Standards with Reservations and not just providing evidence on the promise of the intervention. 


H Century, J., Cassata, A., Rudnick, M., & Freeman, C. (2012). Measuring enactment of innovations and the 


factors that affect implementation and sustainability: Moving toward common language and shared conceptual 
understanding. Journal of Behavioral Health Services & Research, 39(4), 343-361. doi: 10.1007/s11414-012- 
9287-x 


Meyers, D. C., Durlak, J. A., & Wandersman, A. (2012). The quality implementation framework: A synthesis of 
critical steps in the implementation process. American Journal of Community Psychology, 50, 462—480. 

Durlak, J. A. (2015). What everyone should know about implementation. In J. A. Durlak, C. E. Domitrovich, R. 
P. Weissberg, & T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp. 
395-405). New York: Guilford. 


Abry, T., Hulleman, C. S., & Rimm-Kaufman, S. E. (2014). Using indices of fidelity to intervention core 
components to identify program active ingredients. American Journal of Evaluation, published online 
November 26. doi: 10.1177/1098214014557009 


Starting in the FY 2013 competition, ED added a selection factor on the extent to which the evaluation plan 
clearly articulates the logic model and measurable fidelity threshold. 
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The Abt Team’s criteria assess whether the evaluation included: 


e a logic model with key components of the intervention, at least one mediator, and at least one student 
outcome; 


e aset of measurable indicators of implementation fidelity of the key components, a way to combine 
the indicators into a score for each key component, and a threshold that defines adequate fidelity; and 


e the periodic measurement of fidelity. 


The criteria assess only the presence or absence of these elements of the implementation evaluation, not 
their validity or quality. The Abt Team did not assess whether the key components accurately describe 
how the intervention is delivered, nor whether they constitute a full representation of the intervention 
model. 


Each grantee/evaluator team established their own thresholds for adequate fidelity of each of the key 
components in their logic model. The Abt Team provided support, but ultimately the thresholds were 
established by each evaluator/grantee team and reflect their definition of what it means to implement the 
funded intervention with adequate fidelity. This resulted in substantial variation across grantees in the 
way they defined adequate fidelity. 


Our criteria for implementation evaluations were developed in light of the initial needs assessment of the 
grantee plans (described in Section 3.2.1), which revealed that none of the evaluation plans included logic 
models or plans to systematically measure fidelity. We designed the criteria to incentivize evaluators to 
meet minimal standards for the presence of these elements, which would represent a substantial 
improvement over the implementation evaluations as proposed. Though there is a growing consensus in 
the field that evaluators should develop logic models and measure fidelity of implementation, there is 
disagreement about what constitutes valid measurement of and thresholds for adequate fidelity. Therefore, 
we developed criteria that reflect minimum standards for what is generally accepted in the field. 


3.2 Support the Design and Conduct of the i3 Evaluations 


Our technical assistance to evaluators included multiple activities with the common goal of supporting the 
i3 evaluators to design and conduct strong impact and implementation evaluations. All of the technical 
assistance provided to i3 evaluators was aligned with WWC Standards and the additional criteria 
developed by the Abt Team. 


The technical assistance included the following series of activities: 


e An initial assessment by the Abt Team of the quality of the evaluations proposed in the successful i3 
applications; 


e Comprehensive, customized technical assistance for each evaluator, guided by the initial needs 
assessment and aimed at strengthening the design and conduct of the evaluations; 


e Ongoing identification and resolution of risks to the strength of the evaluations while they are being 
conducted; and 


e Periodic assessments of the strength of the evaluations that are shared with both evaluators and ED. 


The technical assistance activities are described in more detail below. 
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3.2.1 Conduct an Initial Assessment of the Need for Support 


Each successful application for i3 funding described a plan for an external evaluation of the proposed 
intervention. The Abt Team conducted a systematic review of these evaluation plans to make an initial 
assessment of the evaluation’s technical assistance needs. Our reviews revealed that none of the 
successful applications provided sufficient information to assess whether it was designed to meet WWC 
Standards. For example, most of the applications proposing quasi-experimental designs (QEDs; more 
than 80 percent) did not discuss how baseline equivalence would be established, which is necessary to 
meet WWC Standards. In addition, many of the proposed designs had significant weaknesses that needed 
to be addressed for the evaluations to be strong. For example, some proposed a randomized controlled 
trial (RCT) in which evaluators planned to exclude students from the analysis based on indicators 
measured after random assignment, such as attendance. 


Our review also concluded that none of the successful applications provided enough information to assess 
the strength of the implementation evaluation. Only 39 percent of the applications included a logic model 
for the intervention, and only 15 percent described methods for assessing fidelity of implementation. 


3.2.2. Provide Technical Assistance to Strengthen Design and Conduct of i3 Evaluations 


The Abt Team used the initial assessment of the funded applications to develop a customized technical 
assistance plan for each of the i3 grantees’ evaluators. The plan included activities for the initial 12-month 
design phase, during which the evaluators finalized the designs for their impact and implementation 
evaluations. 


We delivered the technical assistance primarily through one-on-one calls between the evaluator and a 
technical assistance provider who was a WWC-certified reviewer and experienced in conducting rigorous 
evaluations in schools. The technical assistance provider served as the primary point of contact with the 
evaluator throughout the grant period, from design through analysis and reporting of results. The Abt 
Team also provided standardized tools and templates to help the evaluators track the progress of their 
evaluations (including key design decisions and milestones) and report their findings; and group 
technical assistance (including webinars and in-person sessions at the annual i3 Project Directors 
Meetings). 


Across the 67 grants summarized in this report, the typical i3 evaluator received: 
e 72 calls with a technical assistance provider—approximately monthly during the five-year grant 
period; 


e 4standardized tools/templates to document and track the progress of the evaluation and the analysis 
and reporting of findings for both the impact and implementation evaluations; 


e 5 webinars on the criteria for and the conduct of strong evaluations, delivered during the first year of 
the evaluations; and 29 evaluation-related sessions at annual i3 Project Directors Meetings (2011- 
2015). 


Each of the three technical assistance modes is described in more detail below. 


One-on-One Evaluation Technical Assistance 

One-on-one evaluation technical assistance was ongoing, frequent, and customized. Throughout the grant 
period, each evaluator had regular telephone calls with an evaluation technical assistance provider. The 
calls occurred twice a month during the first year of the grants, when evaluators were revising and 
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finalizing their evaluation plans, and monthly for the remainder of the grant period. The technical 
assistance provider customized the call agendas to the context, resources, key research interests, and 
needs of each evaluation. During the design year, one-on-one technical assistance focused on supporting 
evaluators in designing evaluations that could meet WWC Standards. 


Technical assistance providers tracked and documented their one-on-one technical assistance with 
evaluators, which addressed a wide range of topics related to evaluation design. The topics most 
frequently addressed covered how different methodological decisions might affect assessments of the 
strength of the evaluation. For randomized studies, the most frequent issues were related to (1) the 
conduct of cluster RCTs, including how to determine whether a sample included joiners and ramifications 
for the evaluation’s evidence rating; (2) allowable sample exclusions post-randomization; and (3) issues 
related to calculating the amount of sample loss (attrition) at the student and school levels. For QEDs, the 
most frequent issues were (1) selecting the best matching procedures; (2) selecting baseline measures for 
outcomes for which there were no natural pretests; and (3) establishing baseline equivalence in a way that 
was consistent with the WWC Standards. In addition, evaluators frequently asked for assistance 
calculating power for specific designs, particularly for cluster designs and designs using interrupted time 
series. 


The most common issues affecting the assessment of the implementation evaluations were (1) developing 
a fidelity measure that measured all of the key components of the intervention that were shown in the 
logic model; (2) establishing thresholds at the sample level for each key component; and (3) defining a 
relevant sample and schedule for measuring fidelity and reporting of findings that aligned with ED’s 
expectations. 


Defining fidelity measures and establishing thresholds for adequate fidelity proved particularly 
challenging for the grantees and evaluators, because they had limited, if any, objective data on the extent 
to which different levels of the key components were necessary to achieve outcomes. Further, many 
evaluators recognized that the presence or amount of a key component was not by itself sufficient, and 
they wanted to measure the quality of the delivery of the key component. However, operationalizing and 
measuring “quality” of implementation for inclusion in fidelity measures were beyond the scope of the 
implementation evaluations. When included, quality was most often measured through participant self- 
report, such as teacher perceptions of the usefulness of the training. 


The following examples are the Abt Team’s descriptions of the approaches evaluators took to developing 
measures of implementation fidelity and setting their own thresholds for adequate fidelity. 


The Children’s Literacy Initiative used its Validation grant to implement its early literacy model in 38 
elementary schools. The logic model for the intervention included five key components. The grantee 
defined the same threshold for fidelity of implementation for each of the components as 80 percent or 
more of the treatment schools fully implementing the component. Full implementation was defined in 
terms of a set of indicators for each component. As an example of how the grantee defined fidelity of 


implementation, consider its key component of “teacher professional development”: for this 
component, full implementation for an individual teacher was defined as attendance at the summer 
training institute and attendance at three seminars during the year. At the school level, full 
implementation was defined as at least 90 percent of teachers demonstrating full implementation. At 
the sample level, the threshold for fidelity of implementation was at least 80 percent of schools 
demonstrating full implementation. 
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School of One leverages a /earning progression based on the discrete set of skills students must 
master and research-based evidence on the relationships among those skills. The model integrates 
available data about each student, administers an additional diagnostic instrument, and based on that 
information, creates a playlist for each student specifying a unique set of skills to focus on over a 
period of time. The instructional content comes from publishers, software providers, and other 
educational groups across nine instructional modalities (life instruction, live reinforcement of prior 
lessons, live tutoring, small group collaboration, independent practice, virtual computerized instruction, 
virtual live instruction, virtual live tutoring and homework). A /earning algorithm captures and analyzes 
the data from each lesson and recommends to teachers a unique daily schedule for each student that 
they can adjust as necessary. At the end of each day, students take a unique assessment (the playlist 
update) to measure mastery of the skill they studied. Fidelity of the model includes three key 
components: These three key components are: (1) Operational Aspects (Space and training for 
teachers), (2) Required Resources (computers, bandwidth, written materials) and (3) Sufficient Time 
(instructional time for students and planning and professional development time for staff). The three 
key components are defined by 12 indicators, differentially weighted in the score for the key 
components by virtue of being scored on a one-point, two-point, three-point, or five-point scale. A 
total Component Implementation Score is calculated for each key component, and, based on the 
score, a school is categorized as Low, Medium, or High implementation, based on thresholds 
established for each component. For all three key components, fidelity of implementation of the 
component at the program level is “70% of schools in the implementation sample had a score 
indicating High implementation of the indicators for that component”. For example, high fidelity for the 
key component of Required Resources is defined by meeting the threshold on three indicators (a) at 
least 75% of school days with a sufficient number of computers to accommodate all student 
schedules; (b) % of school days with sufficient broadband and network functionality, and (c) % of 
school days with sufficient supply of student forms/ materials prepared. High fidelity on the summary 


Component Implementation Score across the three indicators is a score of 3 and fidelity at the 
program level is defined as at least 4 of the 6 schools receive a score of 3. 


PTA Comunitario seeks to establish successful partnerships between community-based organizations 
and schools with minority and low-income families in order to increase student achievement, 
particularly of students who are low-income, Hispanic, and/or limited English proficient (LEP). The 
project seeks to establish PTA Comunitarios; establish a partnership between PTA Comunitarios and 
the local school district through training and technical assistance; and facilitate collaborative 
educational leadership projects for PTA Comunitarios and partner schools. The model incorporates 
four key components: (1) Community Organizing and Mobilization; (2) Initiation and Relationship- 
Setting; (3) Taking Collaborative Action for Student Success, Education Leadership Projects; and (4) 
Instituting Sustainable Connections. These four components are defined by nine indicators, each 
scored on a two-point or a three-point scale. The total score across indicators for each key component 
is converted into a Component Implementation Score (Low, Moderate, or High). Depending on the 
component, fidelity of implementation of the component at the program level was defined as “80% of 
the 24 PTA Comunitarios (partnered with 24 schools) had a Component implementation Score = 
High”. For example, fidelity for the key component of Collaborative Educational Leadership Projects is 
defined by (a) whether or not parents in the Comunitario develop a project that addresses a need 
identified by the school (0 or 1); (b) whether at least 80% of parents in the Comunitario participate at 
least monthly in the project (0 or 1); (c) whether at least 80% of parents in the Comunitario rate the 
project as effective (Score on a 5-point scale) in addressing the school need (score of 0 if rating — 0, 
score of 1 if rating = 1-3, score of 2 if rating - 4 — 5). Adequate fidelity on the summary Component 
Implementation Score across the three indicators is a score of 4, and fidelity at the program level is 
defined as at least 20 of the 24 PTA Comunitarios receiving a score of 4. 
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Standardized Tools and Templates 


The Abt team developed four standardized tools and templates for the i3 evaluators: (1) the Study Design 
Template to fully describe impact and implementation evaluation design plans, with embedded supports 
and links to relevant WWC Standards and i3-specific criteria; (2) the Contrast Tool to document each of 
the planned impact analyses, including research questions, samples, outcomes and baseline measures, and 
timing of the analysis; (3) the Fidelity Measure Tool to describe the methodology for measuring the 
fidelity of implementation of the key components of the intervention’s logic model; and (4) Reporting 
Templates that include all the information necessary to review the strength of and summarize the findings 
from the evaluations to guide evaluators in drafting their public reports. 


Group Technical Assistance 


The Abt Team provided group technical assistance to evaluators on technical issues that were common 
across the i3 evaluations, using webinars and conference presentations. This group technical assistance 
comprised more than 100 presentations focused on many of the same topics as the one-on-one technical 
assistance—WWC Standards, i3-specific criteria for providing evidence on the promise of the 
intervention, i3 criteria for implementation evaluations, and evaluation design topics such as multivariate 
matching methods and calculating statistical power. 


3.2.3 Identify and Resolve Risks to the Strength of the Evaluations 


The Abt team identified potential risks to the strength of the evaluations that arose during design and 
during the conduct of the evaluations. The Abt Team developed both a categorization of different types of 
risks and a process for identifying and monitoring the risks. Risks included the following: 


e Issues that could affect the strength of the impact evaluation (e.g., non-random movement of 
evaluation participants after random assignment, or a QED without acceptable baseline measures); 


e Issues that could affect the interpretation of impact findings (e.g., low power to detect effects of the 
intervention); 


e Issues that could affect whether the implementation evaluations could provide high-quality data (e.g., 
absence of a systematic measure of the fidelity of a key component or a threshold for adequate 
fidelity); and 


e Issues that threatened the independence of the evaluation (e.g., the intervention developer planned to 
collect student outcome data). 


The Abt Team reported these risks to ED along with recommended actions to resolve the risk, as 
applicable. During the conduct of the 67 evaluations included in this report, the Abt Team reported 136 
risks to ED. Of the 67 evaluations summarized in this report, 46 evaluations (69 percent) had at least one 
risk reported. Of these 136 risks, 65 (48 percent) were resolved by the evaluator and 71 (52 percent) were 
documented as permanent. 


3.2.4 Periodic Assessments of the Strength of the Evaluations and Reports to the Evaluators 
and ED 


As described in Section 3.2.1, the evaluation plans included in the applications were not detailed enough 
to assess the strength of the evaluation designs or to plan technical assistance activities. Therefore, the 
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Abt Team asked evaluators to submit a complete evaluation plan late in the first grant year (October of 
the year following grant award).'° We provided evaluators with templates for both impact and 
implementation evaluation designs (those described in Section 3.2.2) to ensure the plans contained all of 
the elements necessary to assess the strength of evaluations. 


Abt Team members who were all WWC-certified reviewers, but separate from those providing technical 
assistance, conducted systematic reviews of the strength of each evaluator’s impact and implementation 
evaluation designs. 


This review informed evaluators: 


e in advance how their evaluations would be reviewed and the consequences of different design 
decisions for their potential evidence rating; and 


e about aspects of their designs that represented risks to the potential for their studies to meet WWC 
Standards. 


The Abt Team communicated the results of our reviews to evaluators in detailed feedback memos, 
recommending revisions to the designs that could address these risks. We encouraged them to work with 
their technical assistance provider to revise their evaluation plans to address the issues and resubmit their 
revised evaluation plans for a second (or third) round of review. Our re-reviews assessed the implications 
of the changes for the potential evidence rating. Evaluators could submit changes to their plans until they 
began their data collection. 


Once the evaluation plans were finalized, the Abt Team’s review of the evaluation plans revealed that: 


e 54 of the 67 impact evaluations (81 percent) were designed with the potential to Unofficially Meet 
WWC Standards with or without Reservations. 


e All 19 Scale-up and Validation evaluations and 35 of the 48 Development evaluations (73 percent) 
were designed with the potential to Unofficially Meet WWC Standards with or without Reservations. 


e 65 of the 67 implementation evaluations (97 percent) were designed to provide high-quality data on 
implementation fidelity. 


3.3. Collect Data on the Conduct of and Findings from the i3 Evaluations 


The Abt Team was tasked with summarizing the findings from the completed i3 evaluations, and for 
putting the findings in the context of the strength of the evaluations (see Chapter 4). Therefore, we 
conducted a systematic data collection of information about the conduct of and findings from the 67 
completed evaluations. 


In this section, we discuss how we (1) collaborated with evaluators to prioritize the impact evaluation 
findings, (2) identified the findings from prespecified analyses from among all the impact findings 
reported, and (3) collected the findings of the impact and implementation evaluations. 


'° ED added the submission of a revised evaluation plan as a grant requirement in the cooperative agreements in 


FY 2011 and beyond. 
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3.3.1 Prioritizing the Impact Evaluation Findings 


The i3 impact evaluation plans specified by the evaluators were often ambitious. In addition to their plans 
to estimate the impact of the intervention on student academic outcomes, most evaluators (48 of 67, or 72 
percent) also included other analyses to, for example, estimate the impact of the intervention on teacher 
and school outcomes (see Section 2.3.2 for examples) or on other student outcomes (see Section 2.3.3 for 
examples). 


Evaluations that plan to estimate impacts of the intervention on many outcomes are at increased risk for 
producing a false positive or negative impact. That is, the more impacts estimated by the evaluation, the 
higher the likelihood that any one analysis will yield a statistically significant finding simply by chance. 
The Abt Team provided guidance to evaluators on how to minimize this risk by prioritizing their impact 
analyses and prespecifying a subset of analyses that they would use to assess whether or not the 
intervention was effective (their “confirmatory” analyses).”’ Our guidance was aimed at helping 
evaluators understand how standard methods for accounting for the risk of conducting multiple analyses 
would affect the interpretation of their findings, while at the same time acknowledging that the evaluators 
were ultimately free to conduct and report all analyses in their publicly available reports. 


The Abt Team used the submission of the revised evaluation plan as an opportunity to ask the evaluators 
to establish, in advance, which of their findings the Abt Team would include in this report to draw 
conclusions about whether or not the intervention had positive impacts. We also asked them to commit to 
reporting the results of those prespecified analyses to us at the end of the evaluation. To be considered 
established in advance, the evaluator had to prespecify the analyses prior to collecting the outcome data 
that would be used in the analyses. Therefore, an analysis is considered prespecified if the plan for 
conducting it was established before the evaluator could have begun the impact analysis. 


Virtually all i3 evaluators (66 of 67, or 99 percent) prespecified impact analyses prior to collecting 
student outcome data. One evaluator did not prespecify any impact analyses. Although this evaluator 
developed a plan for estimating the impacts of the intervention on student outcomes, it was not finalized 
prior to collecting student outcome data. 


This process of prioritizing and prespecifying analyses meant that both the evaluators and the Abt Team 
had shared expectations about which findings would be used by the Abt Team to draw conclusions about 
the impacts of the intervention on student academic outcomes. The process set the expectation that 
evaluators would report the findings from the analyses to the Abt Team no matter the result, minimizing 
the extent to which evaluators suppressed null or unfavorable findings when they reported findings to us 
(sometimes referred to as the “file drawer” problem). In addition, the process limited the extent to which 
evaluators could redefine aspects of their analyses in order to find more favorable results to submit to the 
Abt Team for this report (sometimes referred to as “fishing”). 


The findings included in the main body of this report (Chapter 4) are limited to the prespecified analyses 
that focus on student academic outcomes, as the goal of i3 is to fund interventions that improve those 


°0 The Abt Team’s guidance was informed by the WWC review protocols, which establish outcome domains 


within which statistical adjustments should be made to minimize the risk of concluding, by chance, that the 
intervention had a positive or negative impact. 
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outcomes.” It is important to note that although all grantees are required to make the findings from the i3- 
funded evaluation publicly available, there was no requirement regarding which findings grantees should 
report publicly. Therefore, our summary of findings in this report might differ from other public reports of 
findings. 


3.3.2 Identifying a Finding as the Result of a Prespecified Analysis 


The evaluator defined four key elements of each prespecified analysis: (1) the treatment condition, (2) the 
control condition, (3) the educational level of the students, and (4) the outcome domain. Changing one of 
these four elements substantially redefines the impact being estimated; therefore, a finding is considered 
the result of a prespecified analysis if none of those four key elements had changed. The evaluators 
specified detailed analysis plans in advance that included, for example, equations for the regression 
models used to estimate impacts, an approach to including covariates in the impact model (i.e., whether or 
under what conditions they will be included) and plans for addressing missing data. The Abt Team used 
these plans to guide ongoing TA on the conduct of the evaluation but did not assess the extent to which 
the analyses conducted adhered to the details specified in the analysis plans. 


Among the 66 evaluations (of 67) that prespecified their impact analyses, evaluators prespecified 222 
analyses estimating the impacts on student academic outcomes and reported findings from 213 of them 
(96 percent). The vast majority of the 67 evaluators (59, or 88 percent) reported findings for all of the 
prespecified impact analyses; 7 evaluators (10 percent) reported findings for some but not all of their 
prespecified analyses. These 213 findings are the focus of the summary found in Chapter 4 and Appendix 
C, and information on each is listed in Appendix D. 


Although the data collection focused on the findings resulting from prespecified analyses of student 
academic outcomes, we collected all findings that evaluators were willing to submit. Twenty-one 
evaluations (31 percent) prespecified 113 analyses on other outcomes and reported findings from 96 of 
them (85 percent; see Appendix E). Forty evaluations (60 percent) reported 210 findings from analyses 
estimating the impacts on student academic outcomes that were not prespecified (see Appendix F). 


3.3.3. Using a Survey to Collect Information about the Conduct of and Findings from the Impact 
and Implementation Evaluations 


The Abt Team asked the evaluators to complete an online survey to gather all of the information 
necessary to assess the strength of and summarize the findings from the impact evaluations. The survey 
also gathered the information necessary to summarize the findings from the implementation evaluations 
about whether the key components of the intervention were implemented with adequate fidelity. 


To reduce burden on the evaluators, a WWC-certified member of the Abt Team facilitated data collection 
by using existing data sources to complete the survey. Specifically, if evaluators chose to use the 
standardized tools and templates provided by the Abt Team for tracking the progress of their evaluation 
(see Section 3.2.2), those resources were used to populate the survey. If the evaluators used their own 
reporting templates or had completed a draft report, the Abt Team used those resources to populate the 
survey. We then asked evaluators to verify the information populated by the data collectors, and to fill in 
any missing information. 


*1__ U.S. Department of Education, (2010), p. 12072. Findings from prespecified analyses of non-student outcomes 


and student non-academic outcomes are not included in Chapter 4, but are reported in Appendix E. Findings 
from analyses that were not prespecified are reported in Appendix F. 
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4. Did the i3-Funded Evaluations Meet the Goals of 13? 


The short-term goal of the i3 program is to produce strong evaluations. 
The long-term goal is to build evidence on effective interventions at 
increasing scale. In this chapter, we summarize the strength of and 
findings from the 67 i3 evaluations completed by May 2017. 


The short-term goal of the i3 program is met when: 


e The implementation evaluation is high quality (Section 4.1.1); 


e The impact evaluation is independent, meets WWC Standards,” ~° 


those served by each i3 intervention (Sections 4.1.2—4.1.4); 


and adequately represents 


The long-term goals of the i3 program are met when: 


e The implementation evaluation finds that the i3-funded intervention was implemented with adequate 
fidelity (Section 4.2.1); and 


e The impact evaluation finds that the i3-funded intervention had at least one positive, statistically 
significant impact on a student academic outcome (Section 4.2.2). 


Sections 4.1.5 and 4.2.3 provide cumulative assessments of the extent to which the i3 program met its 
goals. 


4.1 Were the i3 Evaluations Strong? 


The i3 program expected grantees to fund strong evaluations that included (1) a high- 
=, quality implementation evaluation and (2) an independent impact evaluation that (3) 
meets WWC Standards™ and (4) adequately represents those served. In this section, we 
summarize whether the 67 completed i3 evaluations met these four expectations; 
Appendix C provides the assessments separately for each evaluation. 


*2 It is important to note that the evidence assessments we conducted only give an evaluation an unofficial WWC 


rating (see Section 4.1.3 for more details) 


*3 At the time of the report, the reviews in this report were conducted under WWC 3.0 Standards and Procedures 


for Group Designs, which covered randomized control trials and quasi-experimental designs; final standards for 
Regression Discontinuity (RD) Designs were not available under WWC 3.0 when these reviews were 
conducted. Therefore, results from an RD study that was conducted by Reading Recovery’s evaluator, are not 
included in this report. Results from the RCT study that was also conducted by the evaluator of Reading 
Recovery are among the studies summarized in this report. 


*4 It is important to note that the evidence assessments we conducted only give an evaluation an unofficial WWC 


rating (see Section 4.1.3 for more details) 
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4.1.1 Were the i3 Implementation Evaluations High Quality? 


ED expected the i3 evaluations to produce high-quality implementation data.” The Abt Team developed 
criteria to assess whether the i3 implementation evaluations met this expectation. An i3 implementation 
evaluation was considered high quality if it: 


Provided a logic model that identified the key components of the intervention, its mediators, and the 
student outcome domains it is intended to affect; 


Periodically measured the implementation fidelity of each of the key components on the sample that 
received the i3 intervention; and 


Compared the measurements to a threshold set by the developer and evaluator that indicates whether 
adequate fidelity was achieved, and reported the results to the Abt Team. 


Sixty-five of the 67 i3 implementation evaluations were considered high quality. The remaining 
implementation evaluations were not considered high quality because implementation fidelity was not 
measured for the sample that was served by the i3 intervention. All 67 evaluations had complete logic 
models and periodically measured and reported whether adequate fidelity was achieved. 


4.1.2. Were the i3 Impact Evaluations Independent? 


All i3 grantees were required to conduct an independent evaluation.*® The Abt Team developed criteria to 
assess the independence of the evaluations. We considered an evaluation independent if the evaluator 
reported at least one impact finding to the Abt Team for which they confirmed that neither the grantee nor 
the intervention developer collected the outcome data,”’ analyzed the outcome data, or reported the 
finding to the Abt Team.”® 


Almost all of the i3 impact evaluations reported findings that were independent. Sixty-five 
evaluations (97 percent) reported at least one finding that was considered independent (Exhibit 4.1). For 
64 of these evaluations, all of the impact findings reported were considered independent. For one of these 
evaluations, some of the impact findings were not considered independent because the grantee collected 
data for a subset of the outcomes. For another evaluation, none of the findings were considered 


25 


26 


27 


28 


U.S. Department of Education (2010), p. 12086. 
U.S. Department of Education (2010), p. 12077. 


Achievement tests administered by states or districts are considered independent, even if the grantee is a state or 
local education agency. 


The criteria for independence could have included an assessment of whether the grantee or developer selected 
the comparison group in QEDs, since they may have been motivated to select comparison groups that are likely 
to have poorer outcomes than those receiving the intervention. However, this is covered by the WWC’s baseline 
equivalence standards, which guard against selecting a comparison group using methods that differ substantially 
from those used to select the treatment group (see Section 4.1.3). 
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independent because the grantee collected data for all outcomes.” Finally, one of the evaluations did not 
prespecify analyses, and therefore the assessment of its independence is not included in this summary.” 


Exhibit 4.1: Independence of the i3 Impact Evaluations 


(5%) 
1 1 1 @ Yes (all findings independent) 


@ Yes (some findings independent) 
® No (no independent findings) 


@ No prespecified analyses 


64 
(95%) 


4.1.3. Were the i3 Impact Evaluations Strong? 


ED expected the i3 grantees to sponsor strong impact evaluations that support conclusions about whether 
the intervention caused impacts on student academic outcomes. ED defines a strong impact evaluation as 
one that meets WWC Standards with or without Reservations.*’ To meet the standards without 
reservations, the WWC requires that assignment to the study groups (i.e., treatment and control) be 
determined by a random process—that is, the study must qualify as a randomized controlled trial 
(RCT)—and have tolerable levels of attrition bias, as determined by the combination of the overall rate of 
attrition and the difference in attrition rates between the two groups. To meet these standards with 
reservations, quasi-experimental designs (QEDs) and RCTs with high attrition must establish baseline 
equivalence of the two groups on a variable that is highly correlated with outcome. Finally, to meet WWC 
standards with or without reservations, the study must be free from serious confounds and use outcome 


°° The evaluator indicated that the grantee did not analyze the outcome data or report the findings to the Abt 


Team. 


3° However, from our review of the findings reported by this evaluation from analyses that were not prespecified, 


we conclude that those findings are independent. 


3! In the Notice Inviting Applications in 2010, ED expected evaluations funded by Development grants to provide 


evidence on the promise of the intervention (U.S. Department of Education (2010), p. 12075). And though 
evidence of promise can support further development and testing of an intervention, it does not support 
conclusions about whether the intervention caused any observed impacts. Starting with the 2015 Notice Inviting 
Applications, the selection criteria for Development grants awarded points for proposed evaluations with the 
potential to meet WWC Standards. 


Abt Associates i3: Summary of 67 Evaluations 1] pg. 25 


measures that have face validity and adequate reliability, that are not overaligned with the intervention, 
and that were collected in the same manner for the two groups.” 


It is important to note that the Abt Team’s assessment differs from an official WWC review in two 
important ways. First, the data available to the Abt Team differ from the data available to the WWC. The 
Abt Team collected the information necessary for the reviews directly from the evaluators and limited the 
summary of impact findings to the prespecified impact analyses (see Section 3.3). The WWC limits its 
reviews to publicly available reports. Second, only reviews conducted under a WWC contract can 
produce an official WWC rating for an evaluation. Thus, the evidence assessments we conducted for this 
report are not a substitute for a WWC review, and we can only give an evaluation an unofficial WWC 
rating. For these reasons, the official WWC rating of the i3 evaluations can differ from the ratings 
reported here. We include the unofficial ratings in this report because meeting the WWC Standards is a 
key goal of the i3 program and because it is important to place this summary of evaluation findings in the 
context of the strength of the evaluations. 


The Abt Team used the WWC Standards that were current at the time of our assessment (Version 3.0 for 
this report).*’ WWC reviews are guided by a review protocol; the Abt Team developed a protocol to 
guide our application of the WWC Standards to the i3 evaluations.“ The Abt Team assigned unofficial 
WWC ratings to the 213 findings reported by evaluators from prespecified analyses. The possible ratings 
were: 


e Unofficially Meets WWC Standards without Reservations 
e Unofficially Meets WWC Standards with Reservations 

e Unofficially Does Not Meet WWC Standards 

e Unofficially Ineligible for WWC Review 


We summarized the ratings for each finding to assign a single rating for each evaluation. For 53 of the 67 
evaluations, all of the findings for the evaluation received the same rating, which became the rating for 
the evaluation as a whole. For 13 of the 67 evaluations, the rating varied across findings; in these cases, 
we identified the most common (“modal”) rating across the reported findings and assigned that rating to 
the evaluation.” One of the evaluations did not prespecify analyses, and therefore the ratings are not 
included in this summary. ”° 


For more details on the WWC Standards, see http://ies.ed.gov/ncee/wwc/Handbooks. 


3 See http://ies.ed.gov/ncee/wwc/Handbooks. 


4 See Appendix B for the review protocol. 


© An alternative approach would be to identify the highest rating across the reported findings and assign that 


rating to the evaluation. Taking that approach changes the rating for three evaluations; one Scale-up grant 
would be reclassified from Unofficially Meets WWC Standards with Reservations to Unofficially Meets WWC 
Standards without Reservations, one Development grant would be reclassified from Unofficially Does Not Meet 
WWC Standards to Unofficially Meets WWC Standards with Reservations, and one Development grant would 
be reclassified from Unofficially Ineligible for WWC Review to Unofficially Does Not Meet WWC Standards. 


°° However, our review of the findings reported by this evaluation from analyses that were not prespecified 


concludes that the evaluation is Unofficially Ineligible for WWC Review. 
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Almost three-quarters of the i3 evaluations were considered strong because the evaluation received 
a rating of Unofficially Meets WWC Standards with or without Reservations (49 of 67, or 73 
percent; see Exhibit 4.2). Of the 67 evaluations, 22 impact evaluations (33 percent) received a rating of 
Unofficially Meets WWC Standards without Reservations and 27 evaluations (40 percent) received a 
rating of Unofficially Meets WWC Standards with Reservations. 


Exhibit 4.2: Modal Evidence Rating Received by Findings from i3 Impact Evaluations 


All Grant Types 
(67 Evaluations) 


& Unofficially Meets WWC Standards 
without Reservations 

& Unofficially Meets WWC Standards 

(33%) with Reservations 

® Unofficially Does Not Meet WWC 
Standards 

& Unofficially Ineligible for WWC 
Review 

®@ No prespecified analyses 


(40%) 


Scale-up Validation Development 
(4 Evaluations) (15 Evaluations) (48 Evaluations) 


1 
8 (2%) 9 


3 (17%) (19%) 


(25%) 
(27%) 


22 
(45%) 


Exhibit 4.2 also shows that most grants of each type funded strong evaluations: 


e All four evaluations funded by Scale-up grants received a rating of Unofficially Meets WWC 
Standards with or without Reservations; 


e 14 of 15 evaluations funded by Validation grants (93 percent) received a rating of Unofficially Meets 
WWC Standards with or without Reservations; and 


e 31 of 48 evaluations funded by Development grants (64 percent) received a rating of Unofficially 
Meets WWC Standards with or without Reservations.” 


37 The Abt Team reviewed the findings from 16 evaluations sponsored by Development grants that were rated 


Unofficially Does Not Meet WWC Standards or Unofficially Ineligible for WWC Review against criteria we 
developed to assess whether they “provide evidence on the promise of the intervention.” All 16 evaluations 
received a rating of Provides Evidence on the Intervention’s Promise. 
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The evidence ratings tend to be higher for evaluations of Scale-up and Validation grants than for 
evaluations of Development grants. For example, though three-quarters of Scale-up grants and two-thirds 
of Validation grants received the highest rating—Unofficially Meets WWC Standards without 
Reservations—approximately one-fifth of Development grants (19 percent) received this rating. 


Eight impact evaluations, all funded by Development grants, were rated as Unofficially Ineligible for 
WWC Review because they conducted a pre-post comparison of outcomes for an intervention group 
without a comparison group that did not receive the intervention. Nine impact evaluations were rated as 
Unofficially Does Not Meet WWC Standards: three were QEDs that did not establish that the treatment 
and comparison groups were equivalent at baseline, one was an RCT with high attrition that did not 
establish that the treatment and control groups were equivalent at baseline, one was an RCT in which 
participants were non-randomly excluded after random assignment and did not establish that the treatment 
and control groups were equivalent at baseline, and four were QEDs in which the observed impact could 
not be attributed solely to the intervention (i.e., there was a confound). 


4.1.4 Were the i3 Impact Evaluation Samples Adequately Representative of those Served by 
each i3 Intervention? 


An important element of meeting the short-term goal of i3 is to learn about the average impact of each i3 
intervention on the population of students served by each i3 project. The population of students ultimately 
served by each i3 intervention may differ from: (1) the population specified in the funded i3 grant 
application or (2) the population the intervention developer describes as the target population for the 
intervention when implemented beyond the i3 project. The Abt Team’s assessment focuses on the 
population of students ultimately served by each i3 intervention, which differs for each i3 project. 


Estimating the average impact of each i3 intervention on the population served by that intervention 
requires a sample that is representative of that population—and not just a sample that is convenient to 
include in the evaluation. Many i3 evaluations non-randomly excluded some of the schools, teachers, or 
students who received the i3 intervention because they were difficult or impossible to include in the 
evaluation (e.g., schools that received the intervention after the sample was selected, students that 
required one-on-one assessments) or did not take the steps necessary for inclusion (e.g., did not comply 
with the evaluation’s assignment process or with data collection requirements). Therefore, the Abt Team 
assessed whether the exclusions of schools, teachers, or students that received the i3 intervention 
jeopardized the generalizability of the impact estimates to the population served by that intervention. 


The impact evaluation sample is considered adequately representative of the population served by the i3 
intervention if it includes (1) all or arandom sample of the schools, teachers, and students that received 
the i3 intervention or (2) anon-random sample of these schools, teachers, or students that satisfied both of 
the criteria listed below: 


1. Excluded no more than 25 percent of the schools that received the i3 intervention.” *° The non- 


random exclusion of schools threatens the generalizability of the impact estimates because the 


8 An i3 evaluation conducted outside of a school setting must not non-randomly exclude more than 25 percent of 


the sites or localities in which the intervention was implemented. 
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impacts could differ between schools that were excluded and those that were included. Ideally, this 
criterion would document the characteristics of the excluded schools to assess whether the exclusions 
jeopardize the generalizability of the impact estimates. However, many evaluators did not report why 
schools were excluded from the sample. 


2. Excluded no more than 10 percent of teachers or students who received the i3 intervention 
based on variables that are known to be strongly associated with student academic outcomes.” 
These variables include teacher experience, prior student achievement, prior academic performance 
(e.g., grade point average), race and ethnicity, income (e.g., eligibility for free or reduced-price 
lunch), English language proficiency (e.g., English learner status, language spoken at home), and 
special education status. 


Over three-quarters of the i3 evaluations (53 of 67, or 79 percent) were considered to have 
estimated impacts on a sample adequately representative of those served by the i3 intervention 
(Exhibit 4.3). Of those: 


e 29 evaluations included all schools that received the i3 intervention and did not exclude students 
based on factors that are strongly associated with student academic outcomes; 


e 4 evaluations included all schools that received the i3 intervention and excluded less than 10 percent 
of the students based on factors that are strongly associated with student academic outcomes; 


e 19 evaluations excluded less than 25 percent of schools that received the i3 intervention and did not 
exclude students based on factors that are strongly associated with student academic outcomes; and 


e 1 evaluation excluded less than 25 percent of schools that received the i3 intervention and excluded 
less than 10 percent of the students based on factors that are strongly associated with student 
academic outcomes. 


Thirteen impact evaluations estimated impacts on samples that were not considered to be 
adequately representative of the population served by the i3 intervention.*' More specifically: 


e 10 impact evaluations excluded more than 25 percent of the schools that received the intervention; 


e 2 impact evaluations included all schools that received the i3 intervention and excluded more than 10 
percent of the teachers who received and delivered the intervention based on teacher experience, a 
factor likely to be associated with student academic achievement; and 


© Because any threshold is inherently arbitrary, we tested whether lowering the threshold from 25 percent to 10 


percent would substantially increase the share of i3 evaluations that satisfied the sample representativeness 
criteria. For the results from this analysis, see Appendix G. 


“0 If the impact evaluation excluded some of the schools that received the i3 intervention, the 10 percent threshold 


was applied to the teachers and students in the remaining schools. 


4! The assessment of one evaluation is not included in the summary because it did not prespecify any impact 


analyses. However, our review of the findings reported by this evaluation from analyses that were not 
prespecified concludes that the evaluation estimated impacts on a sample adequately representative of those 
served by i3. 
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e 1 impact evaluation included all schools that received the i3 intervention and excluded more than 10 
percent of the students who received the intervention based on disability status, a factor known to be 
associated with student academic achievement. 


The percentage of i3 impact evaluations based on adequately representative samples was smaller for 
Scale-up grants (25 percent) than for Validation grants (93 percent) or Development grants (79 percent; 
see Exhibit 4.3). 


Only one of the four Scale-up evaluations included a sample that was adequately representative of the 
population that received the i3 intervention. The lack of representativeness raises concerns about drawing 
conclusions about the impacts on the population served by the i3 interventions from the remaining three 
Scale-up grant evaluations. The evaluation that was considered representative selected a random sample 
of schools that received the intervention and included 82 percent of those schools in the evaluation. The 
other three Scale-up evaluations excluded more than 25 percent of the schools that received the 
intervention. In particular: 


e 1 excluded two-thirds of the schools that received the intervention because they were not sufficiently 
oversubscribed to support random assignment, opened after the third year of the grant, or lacked the 
data needed for the evaluation; 


e 1 excluded all middle and high schools that received the intervention, thereby excluding the 64 
percent of teachers who were in grades 6—12;” and 


e 1excluded more than 90 percent of schools that received the intervention at a discounted price under 
the i3 grant, and only included those schools that received the intervention at no cost. 


All but one of the 15 evaluations of Validation grants (93 percent) were based on samples that were 
adequately representative of the population served by i3. The evaluation that was not based on an 
adequately representative sample excluded more than 25 percent of the schools that received the 
intervention. 


Thirty-eight of the 48 evaluations of Development grants (79 percent) were based on samples that were 
adequately representative of the population served by i3. Thirty-five included at least 75 percent of the 
schools and did not exclude students based on factors that are strongly associated with student academic 
outcomes (21 of these 34 evaluations included all schools). The other three included all schools and 
excluded less than 10 percent of the students based on factors that are strongly associated with student 
academic outcomes. Nine evaluations were not based on an adequately representative sample: six 
excluded more than 25 percent of the schools, and three excluded more than 10 percent of students or 
teachers based on factors that are strongly associated with student academic outcomes (all three of these 
evaluations included all schools). 


“ This evaluation also excluded all teachers placed in participating schools after the first two years of the grant. 


“8 See Appendix B for the review protocol; the assessment for one development grant was not included in this 


summary because it did not prespecify any impact analyses. 
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Exhibit 4.3: Representativeness of the i3 Impact Evaluations 


All Grant Types 
(67 Evaluations) m Yes 


™No 


@ No prespecified analyses 


53 
(79%) 


Scale-up Validation Development 
(4 Evaluations) (15 Evaluations) (48 Evaluations) 


4.1.5 Summary: Did the i3 Evaluations Meet the Short-term Goal of i3? 


To meet the short-term goal of i3, the i3 evaluations were expected to produce high-quality 
implementation data, be independent, meet WWC standards, “ and be adequately representative of those 
served by the intervention. Meeting the short-term goal of i3 requires meeting all four expectations; in 
this section, we summarize whether the i3 evaluations meet each expectation, given that they met the 
prior expectations. The assessments shown in Exhibit 4.4 build on each other, such that, for example, the 
number of grants shown as having an independent evaluation (second column) were also found to have a 
high-quality implementation evaluation (first column). See Appendix H for the assessments for each 
grant. 


e Almost all i3 evaluations (65 of 67, or 97 percent) conducted a high-quality implementation 
evaluation. Meeting the expectations for implementation evaluations was not a common barrier to 
meeting the short-term goals of i3. 


“4 It is important to note that the evidence assessments we conducted only give an evaluation an unofficial WWC 


rating (see Section 4.1.3 for more details). 
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e Sixty-three i3 evaluations (94 percent) were also considered independent; meeting this 
expectation was only a barrier to meeting the short-term goals of i3 for two grants.” 


e Almost three-quarters of the i3 evaluations (48 of 67, or 72 percent) also included strong impact 
evaluations that produced findings that received a rating of Unofficially Meets WWC Standards with 
or without Reservations. Of the 63 grants that met the first two expectations, 15 did not meet the 
expectation that the findings unofficially meet WWC standards, making this expectation the most 
common barrier to meeting the short-term goals of i3. 


e Sixty percent of the i3 evaluations (40 of 67) also conducted the impact evaluation on a sample 
that adequately represented those served by the intervention, thereby meeting all expectations 
necessary to meet the short-term goal of i3. This includes one of the four Scale-up grants, 13 of the 
15 Validation grants, and 26 of the 48 Development grants. 


Exhibit 4.4: Number and Percentage of Evaluations that Meet the Short-Term Goal of i3 


Expectations for Meeting the Short-Term Goals of i3 


Impact Evaluation Number of Evaluations 
High-Quality Adequately Meeting All 
Implementation Independent Unofficially Meets Representative of Expectations 
Evaluation Impact Evaluation | WWC Standards Population Served (N=67) 
v = = - 65 (97 percent) 
v v = - 63 (94 percent) 
v v v - 48 (72 percent) 
v v v v 40 (60 percent) 


4.2. What did the i3 Evaluations Find? 


Meeting the i3 program’s long-term goal of building evidence on effective interventions 
requires identifying those interventions that can be implemented with adequate fidelity 
and are shown to improve student academic outcomes. In this section, we summarize the 
evaluations’ findings; the findings from each of the i3 implementation and impact 
evaluations are presented in Appendix C. 


4.2.1 Did the i3 Implementation Evaluations Find that the Interventions were Implemented with 
Adequate Fidelity? 


The i3 grantees were expected to implement the i3-funded interventions and serve students to improve 
student academic outcomes. As discussed earlier in the report, the evaluators, in consultation with the 
intervention developers, established thresholds for what they considered the adequate implementation of 
each key component named in the intervention’s logic model. As a result, the fidelity measurement 
systems and thresholds for fidelity varied widely across grantees. (For example, whereas one fidelity 
measurement system might focus on the delivery of training to teachers by the program staff, another 


“© The assessments for one impact evaluation are not included in this summary because it did not prespecify any 


analyses. 
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might also include the extent of teacher participation in or satisfaction with the training as well.) 
Regardless of the measurement system, all i3 evaluators were expected to similarly document and report 
the extent to which each key component was implemented with fidelity, relative to those thresholds, in 
each year of measurement. The intervention was considered to have been implemented with adequate 
fidelity if the implementation of the majority of the intervention’s key components met the specified 
fidelity threshold in at least 50 percent of the years of measurement. 


Fifty-two of the 67 interventions (78 percent) were implemented with adequate fidelity (see Exhibit 4.5). 
Thirteen interventions were not implemented with adequate fidelity and two did not include a high-quality 
implementation evaluation. There was variation by grant type in whether or not the interventions were 
implemented with adequate fidelity. Seventeen of 19 Scale-up and Validation implementation evaluations 
(89 percent) found that the interventions were implemented with adequate fidelity compared to 35 of 48 
Development implementation evaluations (73 percent). It is perhaps not surprising that Development 
grant interventions were less often implemented with adequate fidelity, given that the interventions were 
generally earlier in their development and testing at the time of the grant award. 


Exhibit 4.5: Summary of Whether the i3 Implementation Evaluations Found that the 
Interventions Were Implemented with Adequate Fidelity 


All Grant Types 
(67 Evaluations) 


@Yes 
No 


® No Evidence 


Scale-up Validation Development 
(4 Evaluations) (15 Evaluations) (48 Evaluations) 


13 
4 (87%) 
(100%) 
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4.2.2 Did the i3 Impact Evaluations Find that the i3 Interventions Improved Student Academic 
Outcomes? 


The short-term goal of the i3 program emphasizes the value of strong evidence, even when the results 
include null or negative findings. But ultimately, the long-term goal of i3 is to identify interventions that 
improve student academic outcomes. To conclude that the intervention improved student academic 
outcomes, an i3 impact evaluation must Unofficially Meet WWC Standards and find at least one positive 
and statistically significant impact and no negative and statistically significant impacts.*° Exhibit 4.6 
summarizes whether the i3 impact evaluations found any positive impacts on a student academic 
outcome.*” Appendix C reports the summary findings separately for each i3 evaluation; for each finding 
reported by evaluators from prespecified analyses that Unofficially Meet WWC Standards, Appendix D 
includes the magnitude of the finding (i.e., the estimated effect size in standard deviation units) and 
whether it was statistically significant at the .05 level. 


Twelve of the 67 impact evaluations (18 percent) found a statistically significant positive impact on 
at least one student academic outcome. The interventions that were awarded grants on the basis of 
stronger prior evidence of effectiveness were more likely to improve a student academic outcome: 50 
percent and 40 percent of interventions supported by Scale-up and Validation grants, respectively, 
improved a student academic outcome, compared with 8 percent of interventions supported by 
Development grants. 


There are, of course, different ways to summarize the findings to assess whether the i3 interventions 
improved student academic outcomes. We applied two alternative approaches, and present the findings in 
Appendix I. Both approaches yielded almost identical results to those shown in Exhibit 4.6. We also 
present an analysis of whether the results are sensitive to the inclusion of findings from analyses that were 
not prespecified (i.e., the findings reported in Appendix F). 


Given the broad portfolio of educational interventions funded by i3, we expect the magnitude of the 
impacts reported by the i3 evaluations to vary by outcome domain. Exhibit 4.7 presents the average effect 
sizes reported in the four outcome domains in which i3 produced the most evidence: (1) 31 evaluations of 
student achievement in English Language Arts (ELA); (2) 23 evaluations of student achievement in 


“© Findings that are statistically significant at the .05 level were considered statistically significant. Hypothesis 


tests were adjusted for multiple comparisons when two or more findings were from analyses that shared the 
same outcome domain. This adjustment reduced the likelihood of finding one or more significant impact 
estimates purely due to chance. The Abt Team followed the process described in the review protocol in 
Appendix B. 


“’ This approach to identifying interventions with positive effects is motivated by the WWC’s approach to 


identifying effective interventions. The WWC concludes that an intervention has a statistically significant 
positive effect in a particular outcome domain if at least one finding is positive and statistically significant and 
none is negative and statistically significant (see the What Works Clearinghouse, Procedures Handbook, 
Version 4.0, Table IV.2). Furthermore, a tool provided by the WWC, Find What Works, can be used to identify 
studies with one or more statistically significant and positive findings across outcome domains (see 
https://ies.ed.gov/ncee/wwc/). 
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mathematics; (3) 7 evaluations of student achievement in science; and (4) 5 evaluations of educational 
progress or attainment.“® Exhibit 4.7 shows that: 


Seven of the 31 evaluations that examined impacts on ELA achievement found statistically 
significant effects, all of which were positive. There were eight studies that reported negative 
effects, although none of these were statistically significant. 


Five of the 23 evaluations that examined impacts on mathematics achievement found 
Statistically significant effects, and they were mixed. Overall, roughly half of the evaluations found 
positive effects and the other half found negative effects. A similar number of evaluations found 
statistically significant negative effects (two) as found statistically significant positive effects (three). 


Only one of the seven i3 evaluations that examined impacts on science achievement found a 
Statistically significant positive effect. The remaining evaluations found null effects on science 
achievement. 


Two of five (40 percent) of the i3 evaluations that estimated impacts on educational progress 
found significant effects, and both were positive. The remaining three evaluations found null 
effects, one of which was negative. 


48 


The amount of evidence in a domain was measured by the number of i3 evaluations that estimated impacts on at 
least one outcome in that domain. We included only analyses that Unofficially Meet WWC Standards with or 
without Reservations in Exhibit 4.7. 
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Exhibit 4.6: Summary of Whether the i3 Impact Evaluations Found Statistically 
Significant Positive Impacts on Student Academic Outcomes 


All Grant Types 
(67 Evaluations) 


mYes 

® No (null) 

® No (negative) 

(18%) (@ No Evidence (Unofficially Does Not Meet WWC Standards) 
®@ No Evidence (Unofficially Ineligible for WWC Review) 


® No Evidence (no prespecified analyses) 


Scale-up Validation Development 
(4 Evaluations) (15 Evaluations) (48 Evaluations) 


2 


(50%) 
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Exhibit 4.7: Effect Sizes for i3 Interventions, by Outcome Domain 


English Language Arts Achievement (31 Evaluations) Mathematics Achievement (23 Evaluations) 
1 1 
0.9 0.9 
0.8 0.8 
0.7 0.7 
0.6 0.6 
0.5 0.5 
0.4 0.4 
0.3 0.3 
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Science Achievement (7 Evaluations) Educational Progress (5 Evaluations) 
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4.2.3. Summary: Did the i3 Evaluations Meet the Long-term Goal of i3? 


The long-term goal of i3 is to build evidence for effective interventions; this goal is met when an i3 
evaluation finds that an intervention was implemented with adequate fidelity and had positive impacts on 
student academic outcomes. Exhibit 4.8 summarizes whether the i3 evaluations met both expectations. 


e Almost 80 percent of the implementation evaluations—52 of 67—found that the intervention was 
implemented with adequate fidelity. This suggests that for most of the interventions, poor 
implementation was not an obstacle to meeting the long-term goal of i3. 


e Thirteen percent of the evaluations—9 of 67—found that the intervention had been implemented with 
fidelity and had produced positive impacts, indicating that improving student academic outcomes was 
challenging. The nine evaluations that met the long-term goal of i3 include two Scale-up evaluations, 
four Validation evaluations, and three Development evaluations. 


The percentage of i3 evaluations that met the long-term goal of i3 differed by grant type: 50 percent of the 
Scale-up and 27 percent of the Validation grants met the long-term goal of i3, compared with 6 percent of 
the Development grants. Perhaps this is not surprising given the high expectations for prior evidence that 
grant applicants needed to meet to receive a Scale-up or Validation grant. It is interesting to note that 3 of 
the 12 evaluations that found positive impacts also found that the intervention was not implemented with 
adequate fidelity. See Appendix H for the assessments for each grant. 


Exhibit 4.8: Number and Percentage of Evaluations that Meet the Long-Term Goal of i3 


Expectations for Meeting the Long-term Goal of I3 


Intervention Has Positive, Number of Evaluations Meeting 
Intervention Implemented with Statistically Significant Impacts Expectations 
Adequate Fidelity on Student Academic Outcomes (N=67) 
v - 52 (78 Percent) 
v v 9 (13 Percent) 
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5. Did the Scale-up Grants Meet their Proposed Scale-up Goals? 


A key goal of the i3 program was to bring to scale interventions with strong prior evidence of 
effectiveness at improving student achievement, closing achievement gaps, decreasing dropout rates, 
increasing high school graduation rates, or increasing college enrollment and completion rates. The i3 
program selected established organizations that provided evidence of their capacity (e.g., qualified 
personnel, financial resources, management capacity) to scale up and meet ambitious targets for the 
expansion of effective practices. The targets were defined in terms of the number of students to be 
reached by the intervention. To achieve these targets, Scale-up grantees were expected to scale up their 
interventions to a state, regional, or national level. The i3 program provided substantial funding to Scale- 
up grantees in part due to the level of scale proposed in the application. 


In this section, we describe the extent to which the Scale-up grantees: 


e met their self-established “scale-up goal” (or expansion target);*” and 


e implemented their scale-up mechanism(s) with fidelity. 


5.1 Capacity of the Scale-up Grantees Pre-i3 Award 


The recipients of the Scale-up grants had to demonstrate organizational capacity to scale their 
interventions broadly. The four completed Scale-up grants were awarded to mature organizations that 
were all established at least 20 years ago. The grantees varied in size at the time of grant award. The scale 
at which each grantee was operating in the period just prior to receiving the i3 grant provides context for 
what it meant for that grantee organization to implement “at scale.” 


All of the Scale-up grantees were operating nationally just prior to the i3 award: 


e Jn 2009-10, KIPP operated 82 schools serving 21,000 students in grades K-12, opening 
approximately 10 new schools each year;”” 


e In 2009-10, Reading Recovery served 73,000 grade 1 struggling readers in more than 5,000 schools 
in 40 states, and was declining in numbers of students and schools served in the years just prior to 
Be 


e In 2010-11, Success for All was actively used in 881 schools in 48 states serving nearly 500,000 
students in grades K-6;”* and 


“8 The assessments of whether the Scale-up grantees met their goals are based on information provided to ED in 


the grantees’ Annual Performance Reports and their publicly available implementation reports. All other 
assessments in this report are based on the data provided to the Abt Team by evaluators, as described in 
Section 3.3.3. 


https://www.mathematica-mpr.com/our-publications-and-findings/publications/understanding-the-effect-of- 
kipp-as-it-scales-volume-ii-leadership-practices-at-kipp 


50 


51 


http://www.cpre.org/reading-recovery-evaluation-four-year-i3-scale 


°° http://www.mdrc.org/publication/scaling-success-all-model-school-reform 
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e In 2009-10, Teach for America placed 4,100 new first-year teachers in schools serving 
approximately 160,000 students in grades preK—12 in 27 states and the District of Columbia; it also 
was supporting 3,200 second-year teachers.” 


5.2 Assessing whether Scale-up Grants met their Self-Established Scale-up 
Goals 


The i3 program expected that the Scale-up grantees would have ambitious targets for expansion. All four 
Scale-up grantees established large scale-up goals, both relative to where the organizations were before 
the i3 grant award and in the absolute number of new implementation sites/students they intended to 
reach. 


Three of the four grantees met their scale-up goals. 


KIPP proposed to open 15 to 18 new schools each year, for a total of 75 new KIPP schools by the end of 
the grant period. It established a threshold of 65 schools serving at least 50,000 students as the criterion 
for meeting the goal (just over 85 percent of the total). These goals represented an increase of 80 percent 
over its pre-i3 number of schools and a doubling of the number of students served. KIPP met its scale-up 
goals, opening 66 new schools and reaching more than 59,000 new students. 


Reading Recovery proposed that by the end of its five-year grant, it would have trained 3,675 new 
Reading Recovery teachers to serve more than 67,000 grade 1 struggling readers. The grantee established 
a threshold of 80 percent of these expansion goals as the criterion for having met its scale-up goals. 
Reading Recovery surpassed its goals for the number of Reading Recovery teachers and teacher leaders 
trained (3,747 new teachers) who worked with 62,000 struggling readers (92 percent of the goal). 


Success for All (SFA) proposed to implement the model in 150 to 300 new schools each year for a total of 
at least 1,100 new schools that would serve 550,000 students. These goals represented a doubling of the 
number of SFA schools and students being served by SFA just before i3. Midway through the grant, with 
approval from ED, SFA revised its goals to 760 schools serving 380,000 students. SFA did not meet its 
original or revised scale-up goals, implementing SFA in 487 new schools and reaching 276,000 additional 
students. 


Teach for America (TFA) proposed to place 13,500 new first-year teachers in the first two years of the 
grant, reaching more than 850,000 students.”* These goals represented a 65 percent increase each year in 
the number of teachers placed by TFA just prior to i3. The grantee established a threshold of 75 percent of 
its expansion goals as the criterion for meeting its scale-up goals. TFA met its scale-up goal by placing 
10,838 new teachers in the first two years of the grant. 


°8_ttps://www.mathematica-mpr.com/our-publications-and-findings/publications/implementation-of-the-teach- 


for-america-investing-in-innovation-scaleup 


“TFA also established a goal to train 46—47 percent of first-year teachers and 57—58 percent of second-year 


teachers to earn the rating of “highly effective” in the 2011-12 and 2012-13 school years. TFA met this goal 
(defined as reaching 75 percent of the goal) for 2011-12 but not for 2012-13. 


Abt Associates i3: Summary of 67 Evaluations 1 pg. 40 


5.3. Description of Implementation of the Scale-up Mechanisms 


The Abt Team worked with each Scale-up grantee/evaluator team to develop a logic model that specified 
the grantee’s scale-up goals and the key mechanisms through which it expected to reach them. Evaluators 
submitted these logic models to the Abt Team at the end of the design year, and we identified the 
mechanism that was central to how each grantee expected to expand to reach its scale-up goals. We 
assessed the extent to which the Scale-up grantees implemented the key mechanism for reaching their 
scale-up goals using the evaluators’ public reports or their final Annual Performance Report to ED. 


5.3.1 Scale-up Mechanisms 


The three Scale-up grantees that met their scale-up goals also reported success in fully 
implementing the scale-up mechanisms for expansion. The one Scale-up grantee that did not meet 
its scale-up goals reported challenges in the implementation of their planned scale-up mechanisms. 


KIPP. The key mechanism for scaling up was expanding the pool of trained school leaders to meet the 
demand for principals to lead new KIPP schools and to lead existing KIPP schools that needed new 
principals. To expand the supply of school leaders in the pipeline, KIPP both increased the capacity of its 
five KIPP leadership training programs and expanded the number of assistant principals in KIPP schools. 


e KIPP provided training to 1,166 school leaders in its leadership training programs over the life of the 
grant, which exceeded its goal of 1,000 new KIPP-trained school leaders. 


e KIPP expanded the number of assistant principal/dean positions in its schools during the grant period. 
KIPP created assistant principal positions in 171 of its schools, compared with the creation of 96 
positions in the three years prior to the i3 grant. 


Reading Recovery. The key mechanism for scaling up was to open new training centers and hire new 
teacher leaders at 15 of the Reading Recovery university training partners. The new teacher leaders would 
help recruit, train, and support the new Reading Recovery teachers in the scale-up. 


e Although Reading Recovery reported that it did add 15 new training centers and hired 46 new teacher 
leaders across all of its training centers, it is not clear how many of the 46 teacher leaders were 
associated with the new training centers. Nor is it clear how many of the newly trained Reading 
Recovery teachers were trained through the new training centers. 


Teach for America. TFA proposed three mechanisms for growing its corps of TFA teachers. First, TFA 
planned to scale up its recruitment efforts to create a larger recruitment pool. They proposed to achieve 
this increase by broadening their college recruitment base to include new types of colleges and include 
more graduate students and professionals at early stages of their careers. Second, TFA planned to 
implement new recruitment strategies to respond to an increase in attrition of TFA teachers between 
acceptance and matriculation. The new strategies included identifying and selecting committed candidates 
earlier in their college careers to increase commitment and offering transitional need-based grants to TFA 
teachers to help them with expenses prior to their entry into the TFA program. Third, TFA planned to 
increase its size, expanding into new regions of the country. 


e TFA was successful at implementing its planned scale-up mechanisms. TFA increased its recruitment 
presence and efforts in “emerging market” schools (less selective overall but more diverse and with 
large new markets of prospective applicants) and in Historically Black Colleges and Universities and 
in schools that are members of the Hispanic Association of Colleges and Universities. The increased 
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recruitment targeting older students resulted in an increase in the proportion of new TFA teachers 
who were graduate students or post-college professionals. 


e TFA reported that it was able to recruit more students earlier in their college careers than in the pre-i3 
period. 


e TFA expanded into 14 new regions by the second year of the i3 grant. 


Success for All. The primary scale-up mechanism proposed by SFA was to significantly reduce the cost 
to districts and schools of adopting the Success for All program, since SFA had determined that the cost 
(around $150,000) posed a barrier to scaling up the model. SFA planned to achieve cost savings by 
moving to a district-focused model that uses local, district-embedded coaches rather than coaches from 
SFA. It further planned to make the district model more feasible by recruiting schools in concentrated 
geographic areas so that district coaches could serve more schools. 


e SFA reported that it was not able to adequately implement the key mechanisms for reducing costs. 
SFA was able to implement the district-model in only a small number of districts; of the 29 local 
coaches who participated in the first year of training, only 12 continued into the second year and only 
5 continued once i3 funding for the coaches ended. 


e SFA did not achieve as much geographic concentration within districts or among neighboring districts 
as planned: 70 percent of school districts in the i3 scale-up had fewer than three SFA schools, 
compared with 77 percent in the period prior to the i3 grant. 


5.3.2 Use of Scale-up Funds for Financial Incentives to Support Scale-up 


Two of the Scale-up grantees planned to use i3 funds to offset the costs to scale up sites implementing 
their program. These grantees reported on the extent to which they offered these financial incentives to 
schools or districts. It is not clear whether and how these grantees will be able to continue after the i3 
funding ends to provide financial support to sites interested in adopting the programs but for whom cost is 
a barrier. 


Reading Recovery used its i3 grant to provide subsidies to schools interested in the program to cover one- 
time start-up costs of training new Reading Recovery teachers. Schools and districts that received the 
subsidies were expected to maintain a three-year commitment to implementing Reading Recovery. 


In a fourth-year evaluation report on Reading Recovery, the grantee reported that a lower-than-expected 
percentage of i3-funded districts maintained the three-year time commitment to the program.”° In order to 
receive i3 funds, districts, schools, and teachers were asked to make a commitment to implement Reading 
Recovery for three years. The report noted that many districts declined the opportunity because they were 
wary of assuming responsibility for funding Reading Recovery over the long term. And although willing 
districts signed a Memorandum of Agreement to formally document their commitment to the program, 
some districts did not sustain the commitment to Reading Recovery for three years. The report concluded 
that although the i3 grant made it possible to increase the presence of Reading Recovery in schools, it also 
attracted schools to the program that were not committed to funding it themselves. 


°° May, H., Sirinides, P., Gray, A., & Goldsworthy, H. (2016). Reading Recovery: An evaluation of the four-year 


i3 scale-up. A research report. Philadelphia, PA: University of Pennsylvania, Consortium for Policy Research 
in Education. 
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Success for All used its i3 grant to provide a partial subsidy to schools that covered start-up costs of 
implementing SFA, as an incentive to encourage and facilitate adoption of the intervention. Even with the 
subsidies, schools and districts were required to contribute two-thirds of the cost to implement the 
intervention (around $100,000). 


In its final report,°*° SFA suggested that the higher-than-expected rate at which initially implementing sites 
dropped SFA (25 percent before the end of three years) might be because the financial incentive 
motivated some less-than-committed sites to adopt the program. 


" Quint, J., Zhu, P., Balu, R., Rappaport, S., & DeLaurentis, M. (2015) Scaling up the Success for All mode of 
school reform. Final report from the Investing in Innovation (i3) evaluation. New York, NY: MDRC 
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6. Summary and Lessons Learned 


In this section, we summarize the Abt Team’s support to evaluators and the extent to which the strength 
of and findings from the i3 evaluations met the short- and long-term goals of i3. Then we identify key 
lessons for future grant programs considering using an evaluation requirement to produce evidence to 
identify effective interventions. 


6.1 Summary 


In addition to the $99 million that the grantees spent on evaluation, the i3 program invested 
approximately $30,000 per grant per year (or $10 million total) in technical assistance for the 67 
evaluations summarized in this report. The support included a one-on-one relationship with a technical 
assistance provider who was a WWC-certified reviewer and experienced in conducting rigorous research 
in school settings, group technical assistance, and tools and templates for documenting the evaluation 
design and progress toward conducting a strong evaluation. 


Almost three-quarters (73 percent) of the 67 i3 impact evaluations unofficially met the WWC evidence 
standards. All but two of the implementation evaluations produced high-quality data on fidelity, and 
almost all the i3 impact evaluations were independent. Seventy-nine percent were able to evaluate 
impacts on a sample that adequately represents those served by i3. Overall, 60 percent of the i3 
evaluations met the short-term goal of i3 to conduct strong evaluations of those served by i3. This 
suggests that it is reasonable to expect grantees to conduct strong evaluations as part of a discretionary 
grant program, but that even with access to comprehensive evaluation technical assistance, meeting 
WWC standards and including a representative sample of those served is challenging. 


Seventy-eight percent of the i3 implementation evaluations found that the interventions were 
implemented with adequate fidelity to the program models, and 18 percent of the impact evaluations 
found positive impacts. Overall, nine evaluations (13 percent) met the long-term goal of i3 by finding 
evidence of both adequate fidelity and positive impacts on student academic outcomes. 


The findings from these evaluations can help ED in deciding which educational programs warrant 
additional funding and testing—and highlight the value of strong evidence, regardless of what the 
findings are. They also provide credible evidence to local decision makers considering whether to adopt 
particular interventions and useful information to policymakers contemplating the design of future 
discretionary grant programs. 


6.2 Lessons Learned for Future Grant Programs 


6.2.1 Building on Existing Evidence Standards made Expectations for and Assessments of the 
Impact Evaluations Clear 


Future grant programs that consider using an evaluation requirement face the challenge of defining what 
grantees need to do to meet the requirement. The i3 program referenced the existing WWC evidence 
standards when setting its expectations for the impact evaluations. Because the i3 Notice Inviting 
Applications referred to existing standards, potential applicants could review the WWC standards to 
better understand what would be expected from their evaluations. The standards also served as the 
foundation for the Abt Team’s evaluation technical assistance to help evaluators design and conduct 
evaluations likely to meet those standards. 
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The WWC standards played a central role in our ability to assess of whether the i3 grantees—and thus the 
i3 program as a whole—met the goal of contributing to the evidence base on which policymakers at all 
levels of government can make decisions about how to best educate students. Without established 
standards, the evaluation expectations for i3 grantees would have been less clear, and it would have been 
more difficult to assess whether that goal had been met. 


However, it is important to note that the WWC regularly updates their standards and review processes. 
And while the WWC makes its current standards and review protocols available on their website, i3 
evaluators often found it challenging to determine how these changes would affect the review of their 
evaluations. The support of the Abt Team was intended to help evaluators understand the implications of 
the revisions made to the WWC standards in March 2014 for their specific evaluation. 


6.2.2. Developing Standards to Assess ED’s Additional Expectations Provided More Information 
about the Strength of the Evaluations 


Future grant programs may have additional expectations for evaluation that go beyond the internal 
validity of the impact evaluation, which is what is assessed by the WWC standards. ED had expectations 
of the i3 evaluations beyond whether they met WWC standards that the Abt Team operationalized into 
standards in several areas where existing standards were not available. The expectations included that (1) 
the evaluations would provide high-quality data on implementation, (2) the evaluations would be 
independent, and (3) that the impact evaluations would adequately represent those served by each i3 
intervention. 


We hypothesize that these i3-specific standards improved the i3 evaluations. For example, the Abt Team 
had numerous conversations with the evaluators about how to collaborate with the grantees without 
undermining independence. It is unclear whether evaluators and grantees would have attended to the issue 
of independence without that support. However, the limited time and resources that can be devoted to 
standards development for a single grant program, combined with the challenges of finalizing standards 
before evaluations begin, underscore the value of adopting or adapting established standards before 
funding a grant program with high expectations for evaluations. 


Future grant programs might also consider stronger standards to align with higher expectations. For 
example, as discussed in Section 4.1.1, the standards developed to assess the i3 implementation 
evaluations focus on the logic model and measuring whether the key components of the intervention were 
implemented with adequate fidelity. Future standards could be extended to include whether the 
implementation evaluation measured key aspects of the experiences of those in the comparison group, to 
measure whether the experiences are in fact markedly different from those of the intervention group 
(sometimes called the “treatment-control contrast” or the “service contrast”). 


Finally, conducting evaluations that include a sample that adequately represents those served proved 
challenging; of the 9 out of 67 i3 evaluations that found both adequate implementation fidelity and 
positive impacts, six evaluations included a sample that was representative of those served. Future grant 
programs interested in whether each funded evaluation included a sample that adequately represents those 
served should make this expectation explicit to grantees and consider adopting standards like those 
developed for i3. 
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6.2.3. Future Grant Programs Could Find it Challenging to Assess the Strength of Evaluation 
Designs Submitted after Grant Award 


Grant programs may want to assess the strength of the evaluations funded by the grantees as they are 
being designed and while they are being conducted. These assessments provide the opportunity for 
stakeholders, including the grant program office, the grantee, and the evaluator, to revise the evaluation 
plan and avoid common barriers to conducting strong evaluations. Ongoing assessments can also help the 
program office monitor its own progress towards funding strong evaluations across grants. Indeed, a key 
feature of the i3 program is that two of its short-term Government Performance and Results Act (GPRA) 
reporting measures require an assessment of the strength of the evaluation designs. The Abt Team, with 
oversight from IES, provided these assessments on an annual basis to inform ED’s reporting 
requirements. Future grant programs interested in performance measures that include the strength of 
grantee evaluations will need to ensure they have the capacity to make these assessments. 


Similarly, the i3 program included an assessment of the completed evaluations as part of the long-term 
GPRA performance measures. Future education programs could leverage the existing WWC review 
process for grants that are expected to produce evidence that meets WWC Evidence Standards. However, 
if a program had additional or varied evaluation expectations for grants, such as the expectation that they 
be independent or conduct a high-quality implementation evaluation, or that they would provide evidence 
of effectiveness aimed at informing earlier stages of program development (i.e., evidence that the 
intervention is promising), the program office would need to include a separate process for making those 
assessments. 


Having stricter selection criteria around the strength of the evaluation designs included in the applications 
is one way future programs could begin to address this challenge. As mentioned in Section 3.2.1, the Abt 
Team’s review of the i3 grantee applications found they did not contain enough information to determine 
whether the proposed evaluation designs would meet ED’s expectations for strong evaluations. Over time, 
the i3 program has included stricter criteria regarding the quality of the project evaluation being proposed. 
For example, applicants are now required to include an intervention logic model in the application and to 
specifically discuss the magnitude of the impact on student academic outcomes that the proposed impact 
evaluation is designed to detect (“statistical power”). These requirements were intended to improve the 
evaluation designs described in submitted applications. 


Further, programs should consider requiring grantees to submit revised evaluation designs early in the 
grant period and provide clear expectations for what should be included in those plans. The Abt Team 
asked evaluators in the FY 2010 cohort to submit revised evaluation plans and supported the evaluators in 
strengthening the plans. Starting with the FY 2011 cohort, the i3 program required the submission of 
revised evaluation plans and included detailed expectations for those plans in the cooperative agreements 
signed by each grantee in those cohorts. Requiring that grantees submit revised plans alone may not 
ensure the strength of the plans; the Abt Team worked closely with i3 evaluators to try to improve their 
plans and provided ED with detailed reviews about the extent to which the plans met ED’s expectations. 


6.2.4 Expecting Evaluators to Prespecify Analyses Ensured that Findings from those Analyses 
would be Reported to the Abt Team, Regardless of the Direction and Magnitude 


Grant programs that include an evaluation requirement will want to ensure that the findings from the 
funded evaluations are made public and contribute to the evidence base. However, requiring that grantees 
produce a public report does not ensure that all findings will be reported. The i3 program emphasized to 
grantees and evaluators the importance of reporting all findings from the evaluations, whether they are 
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positive, null, or negative, but grantees may have incentives to include some findings and not others in 
their public reports. Further, grantees may also have incentives to conduct additional analyses if those that 
are originally planned are not favorable, and to highlight results from those additional analyses in their 
public reports when they are positive (“fishing”). 


The Abt Team asked the i3 evaluators to prespecify which of their evaluation findings we would include 
in this report summarizing the impact of the interventions on student academic outcomes. When needed, 
the Abt Team provided technical assistance to help evaluators narrow their planned analyses to those that 
would produce the most credible evidence of impacts on students. This report includes 213 impact 
evaluation findings submitted by evaluators, representing 96 percent of prespecified analyses; that the 
findings vary in their direction and magnitude could indicate that they were not selectively chosen. 


It is important to note that the protections described above are unique to this report’s summary of the i3 
evaluation findings. The findings in this report may differ from those included in reports made publicly 
available by grantees and evaluators (which will be reviewed by the WWC as they are made available) 
because we focused this report on findings from analyses of student academic outcomes that were 
prespecified and reported directly to the Abt Team. The prespecification and data collection functions 
performed by the Abt Team were designed to provide some protection against losing potentially 
important findings from the impact evaluations and against including findings from analyses that were not 
planned in advance. 


Future programs could take advantage of existing education evaluation registries, such as the one 
currently under construction by the Society for Research on Educational Effectiveness (SREE),’” ” and 
require that grantees register their evaluation plans. To fully recognize the benefits of the registration 
process, however, staff monitoring the grants would need to commit to holding grantees accountable for 
conducting and reporting on the evaluations as registered. Policymakers interested in including this type 
of registration and review process in future programs should not assume that evaluators will conduct and 
report findings as they were detailed in the evaluation plans, and will report all findings, regardless of 
whether they are favorable to the intervention, without ongoing support and oversight. 


6.2.5 Testing the Interventions at Scale was a Challenge for Scale-up Grants 


The purpose of the Scale-up grants was to bring interventions supported by strong prior evidence of 
effectiveness to scale and learn about their effectiveness at scale. As discussed in Section 5.3, three of the 
four Scale-up grantees met their goals for scaling up the intervention, and all four implemented their 
intervention on a much larger scale than it had previously been implemented. However, only one of the 
four impact evaluations included a sample that was representative of those served by the i3 intervention 
(see Section 4.1.4). This means that three of the four impact evaluations supported by i3 Scale-up grants 
were not able to test the effectiveness of the i3-funded intervention at scale and limits the conclusions that 
can be drawn from these studies. 


°” ~~ Maynard, R., & Spybrook, J. (March, 2016). A registry of effectiveness studies in education. Paper presented at 


the annual meeting for the Society for Research on Educational Effectiveness, Washington, DC. 


°8 ~The WWC previously maintained a registry but transferred the database to the SREE registry, which is 


scheduled to launch in late 2017. 
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An important constraint in testing the effectiveness of the interventions at scale was that all the 
implementation and evaluation activities needed to be completed within the five-year grant period.” All 
four grantees planned to use the full five years of the grant to meet their ambitious scale-up goals. Further, 
two of the four grants (Teach For America and Success For All) funded multi-year interventions that were 
hypothesized to have the greatest impacts more than one year after a teacher or school begins to 
implement the intervention. To complete the evaluation within the five year grant period, the evaluators 
typically selected a sample of teachers or schools that implemented the intervention early in the grant 
period, before the intervention had been taken to scale. These evaluations provide evidence about the 
impacts of these interventions early in the scaling process, and not their effectiveness at scale. Providing 
evidence of the effectiveness at scale would have required a longer grant period with clear expectations to 
select the evaluation sample to adequately represent the schools, teachers, or students that were part of the 
effort to scale the intervention. 


6.2.6 Relying on Administrative Data was a Challenge 


Grant programs aiming to fund interventions to improve outcomes that are routinely measured (e.g., 
student achievement) can benefit from using that administrative data for evaluation purposes. Using 
outcome data that is collected for other purposes is both cost-effective and reduces burden on participants. 
However, states do not routinely test all subjects in all grades, and evaluators cannot estimate impacts of 
interventions in untested grades. This was a particular challenge for interventions aimed at improving 
science achievement because states typically only administer standardized science tests once in middle 
school and once in high school. Further, heavy reliance on administrative data to measure student 
outcomes is risky when states or districts change their plans for testing or collecting data. For example, 
when California did not field the California Standards Tests in 2014, a number of i3 evaluators had to 
either select a new outcome measure or Measure outcomes in another year (e.g., a year earlier). Future 
grant programs should balance the benefits of using administrative data with the risks evaluations face if 
the availability of those data changes. 


6.2.7. Resources for Future Programs that Require Evaluation 


The Abt Team developed a number of resources to support the i3 evaluators that could be leveraged by 
future programs hoping to provide similar support. ED has established the following link to make 
resources available: http://ies.ed.gov/ncee/projects/evaluationta.asp. Examples of available resources 
developed as part of this project include these: 


e The Evaluation Plan Template, which identifies the key components of an impact evaluation plan and 
provides guidance about the information typically included in each section of a plan for evaluating the 
effectiveness of an intervention. 


e The Contrast Tool, which is an Excel spreadsheet designed to support and supplement the 
development on an evaluation plan. The spreadsheet documents the specific comparisons the 
evaluator will make between treatment and control group outcomes to test program effectiveness. 


e The Fidelity Measure Tool, which documents how adequate fidelity to the intervention model will be 
assessed. 


°° ‘These grants were awarded with FY 2010 funds, which expired at the end of September 2015, and therefore the 


i3 program could not grant any extensions to grants in the FY 2010 cohort. 
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Appendix A: Key Components and Measures of Implementation Fidelity for 67 i3 Interventions 


Intervention Description 
Scale-up 
KIPP 


Key Component Name 


Key Component Domain 


Implemented 
Number of Times with 
Fidelity Was Adequate 
Measured Fidelity? 


KIPP schools seek to engage students and parents 
actively in the educational process, expand the time 
and effort students devote to their studies, reinforce 
students’ social competencies and positive 
behaviors, and dramatically improve their academic 
achievement. KIPP schools are grounded in five 
core principles: high expectations, choice and 
commitment, more instructional time, power to lead, 
and focus on results. 


KIPP Schools® 


Target Leadership Structures and 
Supports 


2 YES 


Reading Recovery 


Reading Recovery is an intensive reading 
intervention for struggling elementary readers. The 
Reading Recovery program targets the lowest- 
achieving 15 to 20 percent of first graders in a 
school. These students receive 12- to 20-week 
cycles of daily, 30-minute, one-on-one Reading 
Recovery sessions. The intervention includes daily 
monitoring of students to ensure that instruction is 
responsive to changes in student achievement and 
needs. The intervention also includes a training and 


60 


Staff Background and Selection 


Select/Evaluate Staff 


4 YES 


Reading Recovery Teacher 
Training and Ongoing PD 


Provide PD for Teachers 


4 YES 


Teacher Leader and Site 
Capacity 


Provide PD for Teachers 


4 YES 


Unlike other grantees, KIPP defined fidelity of implementation based on a single key component that was a total of 7 indicators: KIPP-selected principal, 


KIPP-trained principal, KIPP-approved school design plan, school has licensing agreement to use the KIPP name, school is connected to KIPP network (e.g., 
through KIPP communities of practice or national retreats), school participates in school reviews within first two years, school collects and shares 


performance data with KIPP. 
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Intervention Description 


Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 
Fidelity Was 
Measured 


professional development program for teachers and) One to-one Reading Recovery | Develop/institute New Curriculum & 4 YES 
teacher leaders in order to implement Reading Lessons Matarals 

Recovery instruction. 

Success for All (SFA) 

Success for All (SFA) is a whole-school turnaround |) ecential training Provide PD for Teachers and Other Staff 2 NO 
program that focuses primarily on ensuring that 

every child succeeds in learning to read throughout + | Program materials Develop/Institute New Curriculum & 2 YES 
the elementary grades. The program includes Materials 

professional development for all school staff, a K-6 . 

reading program, assessments, and tutoring. SEA School inputs and Develop/institute New Curriculum & 2 YES 
staff also address non-academic issues, support K-6 implementation Materials 

teachers, and work to develop leadership within 

schools. 

Teach for America (TFA) 

Teach For America (TFA) is a nonprofit organization Selection Select/Evaluate Staff 2 YES 
that seeks to reduce educational inequities through 

the provision of high quality teachers to low-income | Placement Select/Evaluate Staff 2 YES 
schools. The program recruits college graduates and . . 

professionals with strong academic backgrounds Summer Institute (pre-service Provide PD for Teachers 2 YES 
and leadership capabilities to commit to teach for training) 

two years in low-income schools. These participants, | Ongoing Support Target Leadership Structures and 2 YES 
called corps members, typically have no formal Supports 

training in education but receive an intensive five- 

week training from TFA before beginning their first 

teaching job, as well as ongoing support throughout 

their two-year commitment. 
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Intervention Description 
Validation 
Advanced ASSET Professional Development 


Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 
Fidelity Was 
Measured 


Advanced ASSET Professional Development aims 
to improve student achievement and self-efficacy in 
math and science by improving teacher 
effectiveness. The program uses the National 
Science Resources Center's model for STEM 
education programs and uses a “teachers teaching 
teachers” model and two research-based 
approaches to professional development. 


The Baby Family and Child Education (FACE) 
program aims to improve children’s school readiness 
through greater parent involvement prenatal to age 
3. Parents as Teachers (PAT), which runs this 
program, provides for early detection of 
developmental delays and health issues, and parent 
education to help parents understand their role as 
their child's first teacher. The Baby FACE program is 
a targeted home-based intervention for American 
Indian families living on tribal reservations in 
primarily rural areas. The intervention provides home 
visits, routine health and developmental screenings 
for children, parent group meetings, and resource 
referrals as needed. 


ASSET Teacher Professional Provide PD for Teachers 4 YES 
Development-STRUCTURE 
ASSET Teacher Professional Provide PD for Teachers and Other Staff 3 YES 
Development-PROCESS 
Development of a “Professional | Support Staff Collaboration 
Learning Community” School 4 YES 
Culture-STRUCTURE 
Development of a “Professional | Support Staff Collaboration 
Learning Community” School 3 YES 
Culture-PROCESS 

The Baby Family and Child Education (FACE) Program 
Training & Professional Provide PD for Other School or Non- 1 YES 
Development School Staff 
Support & Technical Assistance | Provide Coaching for Staff 2 YES 
Service Delivery Develop/Institute New Curriculum & 2 NO 

Materials 
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Implemented 
Number of Times with 


Fidelity Was Adequate 
Intervention Description Key Component Name Key Component Domain Measured Fidelity? 


Children’s Literacy Initiative (CLI) Program 


The Children’s Literacy Initiative (CLI) program 
provides prekindergarten through third-grade 
teachers with training and coaching in effective 
practices for early literacy instruction. CLI’s coaching | Coaching Provide Coaching for Teachers 3 YES 
and professional development aims to raise student 
reading achievement by helping teachers establish 
and maintain literacy-rich classroom environments 
and develop shared standards of high-quality Principal and school-based Provide Coaching for Administrators 3 NO 
instruction. The CLI program includes four key coach professional development 
features: 1) Providing teachers with literacy and leadership meetings 
resources, including book collections, to create a 
literacy-rich classroom environment; 2) Conducting 
professional development institutes and seminars to 
train teachers in strategies and techniques for 
literacy instruction, followed by classroom- 
embedded coaching to help teachers apply these 
Strategies in the classroom; 3) Identifying one model 
classroom teacher per grade who receives intensive 
coaching and support and in whose classroom 
additional embedded coaching can be provided; and 
4) Educating school leaders on how to leverage CLI 
training to sustain high-quality literacy instruction in 
the school. 

Collaborative Strategic Reading (CSR) 
Collaborative Strategic Reading (CSR) Colorado is a | Teacher knowledge of CSR Provide PD for Teachers 2 YES 
research-based intervention for cross-content-area 
teachers to improve discipline-specific reading 
comprehension. CSR Colorado is built on a 


foundation of reciprocal teaching and combines Leadership knowledge of CSR _| Provide PD for Other School or Non- 
learning strategies, parent outreach, and teacher School Staff 2 YES 


Resources and professional Provide PD for Teachers 


development 3 ee 


Model classrooms for Provide Coaching for Teachers 


professional development 3 YES 


Teacher implementation of CSR | Develop/Institute New Curriculum & 


Materials 2 YES 
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Implemented 
Number of Times with 


Fidelity Was Adequate 
Intervention Description Key Component Name Key Component Domain Measured Fidelity? 


professional development and coaching to improve | Leadership implementation of Target Leadership Structures and 


reading comprehension for students. CSR Colorado | CSR Supports 2 YES 
is intended to help students develop metacognitive 
awareness and learn specific approaches District implementation of CSR Involve Parents/Community Members 2 YES 


associated with enhanced reading comprehension parent engagement activities 
using four strategies: (1) preview—brainstorming 
and predicting or setting purposes for reading; (2) 
click and clunk—monitoring understanding and 
taking steps to figure out unknown words or 
confusing ideas; (3) get the gist—determining main 
ideas and gist statements; and (4) wrap-up— 
generating questions and reviewing key ideas or 
summarizing. 

College Readiness Program (CRP) 


The College Readiness Program (CRP) provides 


: Awards Develop/Institute New Curriculum & 2 YES 
supports to schools in the form of program Materials 
management (e.g., regional director oversight, 
student/school-level data, multi-way performance Program Management Plan for and Support Assessment & 2 YES 
feedback), teacher supports (e.g., lead teachers, Data Use 
team meetings, content leads, summer institutes, : 
trainings), student supports (¢.g., paid exam fees, Student Supports ae New Curriculum & 2 NO 


equipment and supplies, open enrollment), and 
awards (administrator, teacher, and student Teacher Supports Provide PD for Teachers 2 NO 
incentives for meeting AP exam score goals). The 
goal of CRP is to create an environment where 
students can participate in AP courses and receive 
instruction that will lead to increases in the share of 
students who pass AP tests, therefore increasing 
college readiness, the central goal of the CRP 
program. 
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Intervention Description 


College-Ready Writers Program (CRWP) 


Key Component Name 


Key Component Domain 


Implemented 
Number of Times with 
Fidelity Was Adequate 
Measured Fidelity? 


The College-Ready Writers Program (CRWP) is Content of Teacher Professional | Provide PD for Teachers 2 YES 
designed to improve the argument writing of Development 
students in grades 7 through 10 by introducing 
teachers to new instructional practices. Local Duration and Breadth of Provide PD for Teachers 
university-based site affiliates provide professional | Teacher Professional 2 YES 
development to English Language Arts teachers. Development 
The core program features include intensive 
professional development to support classroom Duration and Breadth of Provide PD for Teachers : We 
implementation, use of curricular resources, and Teacher Professional 
formative assessment to improve instruction. By Developmen Qvet2 teats 
changing teacher attitudes and instructional Professional Development Provide PD for Teachers 2 YES 
practices, CRWP aims to improve students’ Strategies 
performance on source-based argument writing 
tasks and to increase students’ use of writing 
processes to compose and refine text and to support 
learning in other areas. 
Diplomas Now 
Diplomas nay seeks i mn around high-poverty Strong Learning Environments | Support Staff Collaboration 2 YES 
high-minority middle and high schools using a (SLE) 
whole-school reform model that aims to dramatically 
increase high school graduation rates. This project Professional Development and | Provide PD for Teachers 2 NO 
unites three organizations—Talent Development, Peer Coaching (PDPC) 
City Year, and Communities in School—and . . 
includes a tiered intervention model that uses Early Curriculum for College Support College/Career Readiness 2 NO 
Warning System data to provide interventions Rengness (2) 
targeted to students’ needs; a staffing model that Tiered Intervention Model (Tl) | Plan for and Support Assessment & D YES 
supports interdisciplinary teacher teams with Data Use 
common planning time; teacher professional 
Student Supports (SS) Select/Evaluate Staff 2 YES 


Abt Associates 


Summary of 67 Evaluations | pg. A-6 


Intervention Description 


Key Component Name 


Key Component Domain 


Implemented 
Number of Times with 


Fidelity Was 
Measured 


Adequate 
Fidelity? 


cere piretPandipest Coney : encom Student Case Management Provide Services Targeting 2 YES 

prepare students for college readiness; additional (SCM) Individualized Learning 

student supports including mentoring and 

afterschool programming; student case Integrated On-Site Support Select/Evaluate Staff D YES 

management; on-site supports to facilitate (OSS) 

implementation of the whole school organization ; : 

reforms; and efforts to engage in family and Family and Community Involve Parents/Community Members 2 YES 

community partnerships that support student Involvement (FC!) 

SUCCESS. Program Staff Training and Provide PD for Other School or Non- 2 YES 
Development (PTD) School Staff 

Enhancing Missouri's Instructional Networked Teaching Strategies (eMINTS) 

enhancing Sissouls Instructional Networked Technology infrastructure Develop/Institute New Curriculum & 2 YES 

Teaching Strategies (eMINTS) is based on four Materials 

underlying research-based components: inquiry- 

based learning, high-quality lesson design, a Technology use Develop/Institute New Curriculum & 2 YES 

community of learners, and technology integration. Materials 

The program provides teachers with two years of 

professional development and supports that include | T@@cher professional pbowide Diet Teachers 3 NO 

monthly classroom visits and a suite of technology Gevelopyent 

tools. In the third year, teachers have access to Administrative support Provide PD for Other School or Non- D YES 

additional, newly created professional development School Staff 

that aligns with Intel's suite of web-based teaching 

tools to expand teachers’ use of inquiry-based Ongoing technology support Provide PD for Teachers 2 YES 

learning. eMINTS is hypothesized to improve 

student engagement and achievement in 

mathematics, communication arts, and 21st century 

skills. 
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Intervention Description 
GO College (An Enhanced Version of Talent Search) 


Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 
Fidelity Was 
Measured 


GO College, an enhanced version of Talent Search, 
is implemented in cities and schools with previous 


participation in the GE Foundation Pilot. The primary 
goal is to develop and provide more intensive and 


extensive college access services for schools that 
serve a high proportion of low-income, first 
generation college-going, and minority students. GO 
College provides additional services through Talent 
Search counselors already in high schools with the 
objective of enhancing the college culture, 


increasing access to data, improving attitudes about 
college, and bolstering students' academic 


engagement and achievement to ultimately increase 
high school graduation and college enrollment. 


Abt Associates 


Conduct focus groups and Plan for and Support Assessment & 

2 NO 
survey data needs Data Use 
Data sharing agreements with Plan for and Support Assessment & 

Bins & 3 NO 

districts implemented Data Use 
Data tool developments & Plan for and Support Assessment & 

2 NO 
updates Data Use 
Learning Community (LC) coach | Provide PD for Other School or Non- 
training and professional School Staff 3 YES 
development linked to data tool 
Data tool usage plan for each Plan for and Support Assessment & 

2 YES 
stakeholder group Data Use 
Foster and monitor data tool use | Plan for and Support Assessment & 

3 YES 
for all stakeholders Data Use 
Asset/strength based support Involve Parents/Community Members 1 NO 
work with students and parents 
Embedded college coaches in Select/Evaluate Staff 

3 YES 
schools 
Data based goal setting with Plan for and Support Assessment & 
school principals and other Data Use 2 YES 
stakeholders 
Provide college/university visits, | Support College/Career Readiness 
and college/career awareness 3 YES 
services 
Fostering whole school (WS) Support College/Career Readiness 
events: Career, college-going 1 YES 
and enrichment 
Fostering math and science Develop/Institute New Curriculum & 3 YES 
events Materials 
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Implemented 
Number of Times with 
Fidelity Was Adequate 
Intervention Description Key Component Name Key Component Domain Measured Fidelity? 
Institute Summer Bridge for Develop/Institute New Curriculum & 
ne : 3 YES 
rising 9th graders Materials 
Academic support for taking Support College/Career Readiness 
rigorous college prep courses 1 YES 
and success 
Tutoring and mentoring services | Provide Services Targeting 
Sad cit : 1 YES 
Individualized Learning 
Data use skill building and Develop/Institute New Curriculum & 
project-based data use for Materials 3 YES 
students 
Hold regular and special Support Staff Collaboration 
; 3 YES 
stakeholder meetings 
Hold special and periodic joint Support Staff Collaboration 
meetings to create and maintain 3 YES 
collaboration 
Distribute college-going Support College/Career Readiness 1 YES 
outreach materials 
Academic year meetings for 9th | Develop/Institute New Curriculum & 
; ; 1 YES 
graders in LCs Materials 
Academic year meetings for Develop/Institute New Curriculum & 1 YES 
10th graders in LCs Materials 
Institute supplemental summer | Develop/Institute New Curriculum & 
service for rising 10th, 11th and | Materials 1 YES 
12th graders 
Engage community to identify Involve Parents/Community Members 
. 3 YES 
with brand 
FAFSA and financial aid Support College/Career Readiness 
2 YES 
knowledge and support 
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Intervention Description 
Leadership Assistance for Science Education Reform (LASER) 


Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 
Fidelity Was 
Measured 


Leadership Assistance for Science Education 
Reform (LASER) is a professional development 
model to transform K-12 science education by 
helping state-, district-, and school-based teams 
create the infrastructure to support high-quality, 
inquiry-based science instruction. The intervention 
relies on a research-based curriculum, differentiated 
professional development, and professional learning 
networks. 


Northeast Tennessee College and Career Ready Consortium 


The Northeast Tennessee College and Career 
Ready Consortium aims to improve high school 
students’ college and career readiness by increasing 
their access to, participation in, and completion of 
advanced courses. It seeks to achieve these goals 
by scaling up local promising practices to offer a 
wide array of advanced high school and college 
credit-bearing courses, particularly in mathematics, 
science, foreign languages, and career and technical 
education. The Consortium uses distance and online 
learning, as well as college partnerships, to increase 
offerings of Advanced Placement (AP), dual 
enrollment, and other upper-level high school 
courses. The i3 grant also was used to create a 
regional coordinating body to analyze course supply 
and demand in the region and determine course 


Science Technology Concepts Develop/Institute New Curriculum & 3 YES 
(STC) Curriculum Instruction Materials 

Research-Based Inquiry Develop/Institute New Curriculum & 3 YES 
Instruction Materials 

Differentiated Professional Provide PD for Teachers 3 NO 
Development 

Consortium management and Provide PD for Teachers and Other Staff 9 YES 
communication 

College and career ready Develop/Institute New Curriculum & 

counselors team promotes a Materials 2 NO 
college-going culture 

Instructional specialists increase | Target Leadership Structures and 

quality of instruction and student | Supports 2 YES 
participation in i3 courses 

Technology coordinator/director | Target Leadership Structures and 

of learning resources ensure Supports 2 NO 
access to distance and online 

courses 

College partners reduce barriers | Develop/Institute New Curriculum & 9 NO 
to dual enrollment courses Materials 
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Intervention Description 

needs; offer professional development for teachers 
in an effort to improve the rigor of courses; and 
provide college and career counseling to encourage 
college access and help students with the college 
application process. 


Abt Associates 


Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 


Fidelity Was 
Measured 


LEA partners share mission and | Support Staff Collaboration 2 YES 
resources, adopt practices in 

their schools, and encourage 

cultural change 

Resources and services provide | Develop/Institute New Curriculum & 

infrastructure to expand and Materials 2 YES 
sustain program capacity 

i3 leadership provides Target Leadership Structures and 

management and Supports 1 YES 
communication 

College and career ready Develop/Institute New Curriculum & 

counselors team promotes a Materials 1 NO 
college-going culture 

Learning resources team Develop/Institute New Curriculum & 1 NO 
increases quality of instruction Materials 

Learning resources team Institute Structural Change 

increases access to courses 1 NO 
through distance and online 

technology 

Learning resources team Institute Structural Change 

expands opportunities for 1 NO 
college level courses 

Resources and services provide | Institute Structural Change 

infrastructure to expand and 1 YES 
sustain program capacity 
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Implemented 


Intervention Description 
Reading Apprenticeship (RA) 


Reading Apprenticeship (RA) is an instructional 
framework that helps teachers support discipline- 
specific literacy and learning. This intervention 
provided teachers with 65 hours of inquiry-based RA 
professional development over the course of 12 
months. The professional development was 
designed to transform teachers’ understanding of 
their role in adolescent literacy development and to 
build enduring capacity for literacy instruction in the 
academic disciplines. Changes in teacher attitudes 
and instructional approaches are hypothesized to 
result in changes in student attitudes and motivation 
while, at the same time, building skills and 
knowledge for subject-specific literacy tasks, 
strengthening students’ view of themselves as 
readers and learners, and yielding substantial gains 
in student achievement. Teachers were also 
supported by teacher leaders who held monthly 
school-based meetings throughout implementation 
and state coordinators who provided support and 
resources. 


Reading Enhances Achievement During the Summer (READS) 


The Reading Enhances Achievement During the 
Summer (READS) program aims to improve reading 
comprehension in elementary school students by 
fostering children's engagement with books over the 
summer. READS addresses the challenge of access 
to books at home by distributing books over the 


Abt Associates 


Number of Times with 
Fidelity Was Adequate 
Key Component Name Key Component Domain Measured Fidelity? 
Teacher Attendance at Provide PD for Teachers 1 NO 
Professional Development 
Teacher Attendance at Monthly | Provide PD for Teachers 3 YES 
Meetings 
Teacher Leader Recruitment Select/Evaluate Staff 3 YES 
Delivery of PD Content Adheres | Provide PD for Teachers and Other Staff 1. YES 
to Standards 
Book Activities Develop/Institute New Curriculum & 2 YES 
Materials 
Classroom Lessons Develop/Institute New Curriculum & 2 YES 
Materials 
READS Family Night Event Involve Parents/Community Members 2 YES 
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Implemented 
Number of Times with 
Fidelity Was Adequate 
Intervention Description Key Component Name Key Component Domain Measured Fidelity? 
pees ments =e nies paltelnae i Summer Parent Follow-up Involve Parents/Community Members 2 YES 
comprehension activities, including both teacher- 
taught lessons at the end of the school year that 
model comprehension activities and independent 
lessons during the summer months. To promote 
family engagement, READS hosts a 90-minute 
family literacy event during which parents learn 
about the program, and intervention staff contact 
parents during the summer if their children have not 
returned completed independent lessons to the 
READS program. 
StartSmart K-3 Plus Program 
The StartSmart K-3 Plus program is an extended Delivery by certified teachers Select/Evaluate Staff 4 YES 
school year intervention that provides 25 days of Focus on literacy and numeracy | Develop/Institute New Curriculum & 
summer educational services to students in grades Materials 3 YES 
Krd to learn core academic content. The MReennon Parent involvement Involve Parents/Community Members 4 YES 
aims to turn around high poverty and persistently : — - 
low-performing schools by improving student Provide 25 additional days of Institute Structural Change 4 YES 
achievement outcomes. Suen ee 
Provide breakfast, lunch, and Institute Structural Change 
; 4 YES 
transportation 
Teacher professional Provide PD for Teachers 
a 4 YES 
development in literacy 
Virginia Initiative for Science Teaching and Achievement (VISTA) 
The Virginia Initiative for Science Teaching and Teacher Professional Provide PD for Teachers 3 YES 
Achievement (VISTA) project provides professional Development 
development for elementary and secondary science 
teachers and evaluates the overall effectiveness of Coach Professional Provide PD for Other School or Non- 3 YES 
VISTA on participating teachers and their students. Development School Staff 
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Intervention Description 

VISTA aims to improve science teaching, through 
intensive professional development, and increase 
student learning throughout Virginia especially in 
high-need (high-poverty, high minority) schools. 
Development 

The Achievement Network (ANet) 


Key Component Name 


Principal Involvement 


Key Component Domain 


Provide PD for Other School or Non- 
School Staff 


Implemented 
Number of Times with 


Fidelity Was 
Measured 


Adequate 
Fidelity? 


3 YES 


The Achievement Network (ANet) is a whole-school 


Aligned Assessments 


Plan for and Support Assessment & 


achieve greater success in the AP classroom. 
Innovative formative assessments, instructional 
strategies and professional learning tools are 
designed around the Challenge Areas of the AP 
course to focus classroom time where it matters 
most. Resources for the program are primarily web- 
based, but there are some opportunities for in- 
person professional development and collaboration 
around the use of resources. 


reform model designed to train teachers and school Data Use 2 NO 
leaders in the use of student interim assessment 
data to inform instruction and curriculum choices. To | Logistical Support Plan for and Support Assessment & 2 NO 
enable teachers and school leaders to imbed data Data Use 
driven decision-making in their everyday practice, : . 9 NO 
ANet provides them with interim assessments and Coaching Provide Ceachingiior Teacnels 
data reports, coaching, and access to peer Network Supports Support Staff Collaboration 2 YES 
networks. 
Advanced Placement (AP) Insight 
The overall goal of the Advanced Placement (AP) — | Online Collaboration Forums Support Staff Collaboration 2 YES 
Insight program is to provide research-based, 
classroom-tested tools such as instructional Online Resources Develop/Institute New Curriculum & 2 YES 
activities, curriculum resources, and formative Materals 
assessments, to support teachers and students to . ; . 

eS Professional Learning Provide PD for Teachers 2 YES 
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Intervention Description 
Advancement via Individual Determination (AVID) 


Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 
Fidelity Was 
Measured 


The Baboquivari Unified School District Wisdom 
Project employs two key strategies—the 
Advancement via Individual Determination (AVID) 
elective course and a comprehensive school-wide 
college-readiness strategy. AVID strategies include 
training teachers and administrators to help students 
succeed in school and prepare for higher education. 
The elective class is offered to selected students in 
each grade (6-12) for one period per day; students 
work on study and organizational skills and receive 
academic assistance from AVID-trained teachers 
and tutors. The school-wide program offers college- 
readiness activities for all students. 


AVID Implementation 


AVID Training 


Support College/Career Readiness 


Provide PD for Teachers and Other Staff 


2 NO 
2 YES 


Enrichment Activities 


Support College/Career Readiness 


2 YES 


Technology Standardization and 
Training 


Develop/Institute New Curriculum & 
Materials 


1 YES 


Around the Corner (ATC) 


Around the Corner (ATC) is a program for both 
preschool and kindergarten students that enables 
teachers to show children how the world works for 
concepts that are not possible to illustrate in a 
classroom with actual objects. The program includes 
computer activities and videos for children. 
Opportunities to view videos again at home (Home 
Links) provide the repeated experiences with 
language concepts and vocabulary that are 
necessary for language development. An additional 
component of the ATC intervention is interactive, 
video-based professional development for teachers. 
ATC is designed to provide a foundation in language 
and literacy, mathematics, science, listening and 
social skills, creative expression, and positive self- 
esteem through a holistic, thematic approach to 
instruction. 


Communication meetings for 
principals and facilitators 


Target Leadership Structures and 
Supports 


2 YES 


Distribution of ATC materials 


Develop/Institute New Curriculum & 
Materials 


2 YES 


Teacher professional 
development 


Provide PD for Teachers 


2 YES 


Train Success for All coaches 
on new media 


Provide PD for Other School or Non- 
School Staff 


2 YES 
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Intervention Description 


Key Component Name 


Arts Achieve: Impacting Student Success in the Arts 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 
Fidelity Was 
Measured 


Arts Achieve: Impacting Student Success inthe Arts | professional Development Provide PD for Teachers and Other Staff 3 NO 
aims to improve student achievement in targeted 
arts disciplines (dance, music, theater, and visual On-site Consultancies Provide Coaching for Staff 3 YES 
arts) and improve students’ writing and English . ; 
language arts skills through targeted professional Classroom Planning and Develop/institute New Curriculum & 3 NO 
development on the use of balanced assessment nies Materials 
(formative and summative) and the use of data to School Planning and Arts Support Staff Collaboration 3 NO 
drive instruction. The intervention translates the Programming 
standards and information from assessments into 
classroom practices that support improved arts 
achievement for all students. The intervention also 
promotes innovations in students’ and arts 
specialists’ access to content and assessment 
feedback through the use of technology. 
Arts for Learning (A4L) Lessons 
ob for Learning (A4l) Besson ts a Supplemental Professional Development for Provide PD for Teachers 1 YES 
literacy program that integrates arts into the Classroom Teachers 
language arts curriculum to raise student 
achievement in reading and writing and to develop | Ongoing Teacher Support Provide PD for Teachers 2 NO 
learning and life skills. ; 
Professional Development for Provide PD for Other School or Non- 2 YES 
Teaching Artists School Staff 
AAL Lessons Delivery by Develop/Institute New Curriculum & 2 YES 
Classroom Teachers Materials 
AAL Artist Residency Develop/Institute New Curriculum & 2 YES 
Materials 
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Implemented 
Number of Times with 
Fidelity Was Adequate 


Intervention Description 


Bay State Reading Institute (BSRI) 


Key Component Name 


Key Component Domain 


Measured Fidelity? 


The Bay State Reading Institute (BSRI) ie whole- Coaching of Principal Provide Coaching for Administrators 4 YES 
school reform model that aims to improve literacy for 

students in grades K-5. BSRI provides professional Material Adoption Develop/Institute New Curriculum & 4 YES 
development and coaching to principals, teachers, Materials 

and school-based reading coaches to help them Saieedonal paula 4 YES 
implement a research-based core curriculum, pment Provide PD for Teachers 

frequent student assessment, differentiated Coaching of School-based Provide Coaching for Staff 4 YES 
instruction, and literacy interventions. The BSRI Literacy Coach 

model emphasizes collaboration among staff in 

planning, teaching, and using data. 

Boston Teacher Residency (BTR) 

The Boston Teacher Residency (BTR) aimsto ———/_Rectructured Residency Provide PD for Teachers 3 YES 
improve student engagement and academic learning 

in Boston Public Schools’ turnaround and low- Concentration of BTR Select/Evaluate Staff 3 NO 
performing schools through teacher and school Graduates in Schools 

a coke dl ae © | induction Support Provide Coaching for Teachers 3 YES 
restructuring of the residency program; (2) Data Support Plan for and Support Assessment & 3 YES 


concentrating BTR graduates in turnaround schools; 
(3) having Clinical Teacher Educators provide 
training and content-specific induction; and (4) 
providing data analysts and products. By 
implementing these key components, BTR expects 
to increase teachers’ professional capacity, create a 
coherent instructional guidance system, and create 
strong instructional leadership, all of which will help 
develop (a) conditions necessary for ambitious 
teaching and instruction that meets the needs of 
diverse learners and (b) an increased number of 
effective teachers. 


Data Use 
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Implemented 
Number of Times with 


Fidelity Was Adequate 
Intervention Description Key Component Name Key Component Domain Measured Fidelity? 


Building Assets, Reducing Risks (BARR) Model 


The goals of the Building Assets, Reducing Risks 

(BARR) model are to increase student coursework 
success and improve achievement. The BARR Restructuring Institute Structural Change 3 YES 
model reduces social/emotional barriers to learning 


Professional development Provide PD for Teachers 3 YES 


thfowats (h profesciovial development tor achers Parent involvement Involve Parents/Community Members 3 YES 
staff, and administrators and (2) restructuring of the | Developmental Assets Develop/Institute New Curriculum & 3 YES 
9th grade blocks served by teacher teams. curriculum Materials 

Fundamental elements of the approach are creating 

positive, intentional relationships (staff to staff, staff | Block meeting review Provide Services Targeting 3 YES 
to students, and students to students); collaborative Individualized Learning 

problem solving by teacher teams; and using Risk review Provide Services Targeting 3 YES 


existing technology platforms and real-time student 


dios ‘ ; Individualized Learning 
data to guide instructional action. 


Whole student emphasis Support Staff Collaboration 3 YES 
Contextual support Target Leadership Structures and 3 YES 
Supports 
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Implemented 
Number of Times with 


Fidelity Was Adequate 
Intervention Description Key Component Name Key Component Domain Measured Fidelity? 


Collaborative Organizational Model to Promote Aligned Support Structures 


The Collaborative Organizational Model to Promote 
Aligned Support Structures (COMPASS) is a district- 
level reform initiative that works to align and better Procedural/Pedagogical Critical / Target Leadership Structures and 2 YES 
train the district's seven types of support staff: Component Supports 
Instructional Facilitators, Instructional Technologists, 
Exceptional Children Specialists, Rt! Coordinators, 
Limited English Proficiency Specialists, Intervention 
Specialists, and Differentiation Specialists. 
Increasing the expertise of support staff is expected 
to lead to teachers receiving higher quality support, 
which then will increase their knowledge, skills, and 
performance. Ultimately, COMPASS's long-term 
goal is to improve the academic achievement of all 
students, with a focus on students with high needs, 
disabilities, and limited English proficiency. 
CollegeYES 


Educative Critical Component —_| Provide PD for Teachers and Other Staff 3 YES 


CollegeYES is all innovative way for schools to offer Middle School: CollegeYES Provide PD for Teachers and Other Staff 3 YES 
a technology certification program to middle school Summer Camp 


students. Students demonstrate technology literacy 
by creating projects that meet the International Middle School: CollegeYES Provide PD for Teachers 4 YES 
Society for Technology in Education (ISTE) National | Professional Development 
Educational Technology Standards for Students 
(NETS*S). A cohort of students in a structured peer- 
mentoring program assist other students as well as 
teachers in implementing the two project-based Middle School: CollegeYES Develop/Institute New Curriculum & 
learning technology projects that are required for Science and College/Career Materials 3 YES 
certification. College YES allows schools to integrate | Project 


opportunities for improved technology literacy into 


Summer Camp 


Middle School: Student Institute Structural Change 3 YES 
Technology Leadership Club 
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Intervention Description 


Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 


Fidelity Was 
Measured 


encourages all students to complete technology : ; ; 
projects that are creative and personally interesting. sai ra ployide Pion Tesenes 5 YES 
High School: Student Institute Structural Change é NO 
Technology Leadership Club 
High School: CollegeYES Develop/Institute New Curriculum & 4 YES 
Science and College/Career Materials 
Project 
Convergence Academies 
Convergence Academies is a whole-school reform — | Connected Learning Supports | Institute Structural Change 2 YES 
model that infuses digital media arts and technology 
throughout the school to engage students in Instructional Framework in Develop/Institute New Curriculum & 2 YES 
challenging, interest-driven learning; improve their Digital Media Materials 
academic achievement; and develop critical 21st ; 
century skills, such as communication, collaboration, Professional Learning Supports | Provide PD for Teachers 2 NO 
problem solving, and creativity. 
Curriculum 2.0 
The objectives oh Cumiculum 2.0, or the Elementary Grades K-5 digital integrated Develop/Institute New Curriculum & 2 YES 
Integrated Curriculum (EIC), were to (1) develop cuimieuliim and sssessnenic ilateriats 
digital K-5 curricula and assessments that integrate 
the four core subjects, along with the arts, around Online learning community for Support Staff Collaboration 2 NO 
critical thinking and academic success skills; (2) all grade levels 
ale ea eee monet . Professional development Provide PD for Teachers and Other Staff 2 YES 
implementation of the integrated curriculum and District and school-level Target Leadership Structures and 2 YES 
assessments; and (3) increase percentages of structures and supports Supports 
traditionally under-represented students performing 
at advanced levels. Montgomery County Public 
Schools designed a staggered implementation of 
Curriculum 2.0, implementing the new curricula one 
grade at a time starting with kindergarten. 
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Intervention Description 
Data-Driven Decision Making and Information Technology Curricula in Schools 


Key Component Name 


Key Component Domain 


Number of Times 
Fidelity Was 
Measured 


Implemented 
with 
Adequate 
Fidelity? 


The Data-Driven Decision Making and Information 
Technology Curricula in Schools project aims to 
explore the effectiveness of a data-driven decision 
making process that incorporates information 
technology and specific content driven interventions 
(in literacy, math, and science) in grades K-12. The 
project is implemented in low-performing schools 
and targets students with low academic 
performance. The intervention includes a sequential 
focus on content areas beginning with a focus on 
language arts in elementary school, continuing in 
middle school with a focus on mathematics, and 
then in high school with a focus on science, 
technology, engineering, and mathematics. 


E3TL Teacher Performance Evaluation System 


Professional Development 


Instruction 


Provide PD for Teachers and Other Staff 


Develop/Institute New Curriculum & 
Materials 


YES 


YES 


Assessment 


Plan for and Support Assessment & 
Data Use 


YES 


E3TL Teacher Performance Evaluation System is an 
initiative to support the implementation of rigorous 
and comprehensive performance-based teacher 
evaluation. The initiative encompasses the 
development of professional teaching standards for 
educators. Additionally, it includes performance 
rubrics and provides professional development to 
teachers around the newly-created standards. 


EngageMe P.L.E.A.S.E. 
EngageMe P.L.E.A.S.E. provides educators in 


middle and high school math and English language 
arts (ELA) classes with an integrated data system of 


Development of 
Training/Materials 


Develop/Institute New Curriculum & 
Materials 


YES 


Training/Ongoing Support 


Provide PD for Other School or Non- 
School Staff 


YES 


Teacher Evaluations 


Integrated Learning Platform 
and Data Management System 
(LMS) 


Select/Evaluate Staff 


Plan for and Support Assessment & 
Data Use 


YES 


NO 
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Intervention Description 


Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 


Fidelity Was 
Measured 


aude prommere inet pens sueenneesan Develop Professional Learning | Provide PD for Other School or Non- 
furnishes relevant learning activities. Teachers have aaa 9 YES 
: : Support Teams: Train the School Staff 
access to a data system that provides them with ; 
: : Trainer 
timely feedback about the current academic 
performance and learning needs of their students, Provide Teacher Professional Provide PD for Teachers 2 YES 
and resources for learning activities aligned to Development 
standards and learning preferences. 
School-Level Support Develop/Institute New Curriculum & 2 YES 
Materials 
District-Level Support Develop/Institute New Curriculum & D YES 
Materials 
Every Child Ready (ECR) 
Every Child Ready (ECR) is a full-day preschool In-Class Coaching Provide Coaching for Teachers 3 YES 
program designed for at-risk children. ECR focuses 
on universal screening, regular progress monitoring, | On-Site Professional Provide PD for Teachers 3 YES 
differentiated instruction based on children’s Development 
progress, specialized support plans for children with 3 YES 
IEPs, and professional development and individual Teacher Education/Degree Select/Evaluate Staff 
coaching for teachers. ECR is theorized to enhance | Classroom Instruction Develop/Institute New Curriculum & 3 YES 
children's language, literacy, and numeracy Materials 
development. 
Everyday Arts for Special Education (EASE) 
Everyday Arts for Special Education (EASE) Professional Development Provide PD for Teachers and Other Staff 2 YES 
provides professional development to special Workshops 
education teachers and is designed to improve 
student achievement in the areas of communication, | Collaborative Classroom Provide Coaching for Teachers 2 YES 
socialization, academic learning, and arts proficiency | Modeling (CCM) 
through integrated, arts-based approaches. ; : 
On-Site Professional Provide PD for Teachers y) YES 
Development (OSPD) 
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Implemented 


Number of Times with 
Fidelity Was Adequate 
Intervention Description Key Component Name Key Component Domain Measured Fidelity? 
Classroom Instruction (Cl) Develop/Institute New Curriculum & 9 YES 
Materials 
Exceptional Coaching for Early Language and Literacy - Enhanced (ExCELL-E) 
The iB-funded nevemon extended and refined the Coaching feedback: Coaches Provide Coaching for Teachers 
Exceptional Coaching for Early Language and review videotapes of teachers’ 
Literacy (EXCELL) preschool professional use of module strategies and 
development model so that it (a) integrated provide feedback that is 1 YES 
technology into the training protocol to allow more comprehensive, direct, and 
teachers and children to be served; (b) addressed congruent w/ ExCELL-E 
the needs of English Language Learners (ELLs) as . 
: : strategies 
well as native speakers of English; and (c) served 
teachers and children in preschool, kindergarten, Progress monitoring: Teachers | Plan for and Support Assessment & 
and first grade. By improving teachers’ instructional | monitor child progress and Data Use 1 YES 
Strategies, particularly those related to language and | adjust instruction 
literacy practices, the intervention aimed to improve : , 
students’ language and literacy outcomes. Teacher PD: Teachers . Provide PD for Teachers ; a 
participate in and master online S 
training modules 
Teacher practice: Teachers use | Develop/Institute New Curriculum & 1 NO 
key techniques with frequency Materials 
and quality in classrooms 
The Expository Reading and Writing Course (ERWC) 
The Eee) Reading and Waiting nee Curriculum Materials Develop/Institute New Curriculum & 1 YES 
(ERWC) is a 12th grade English course developed Materials 
by a task force of California State University faculty 
and high school educators to improve the academic | Professional Development Provide PD for Teachers 1 YES 
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literacy of high school seniors and thereby reduce 
the need for English remediation in college. The 
program is disseminated to schools and English 
teachers through professional development in which 
teachers further improve their pedagogical skills to 
teach the ERWC. 


Facilitating Long-Term Improvements in Graduatio 


The goal of Facilitating Long-Term Improvements in 
Graduation and Higher Education for Tomorrow 
(FLIGHT) is to increase the extent to which low- 
income students with academic promise are 
prepared for, enrolled in, and successful in college. 
Prepaid student scholarships are a key input to the 
FLIGHT model; each participating student is 
guaranteed a four-year college scholarship provided 
he/she maintains good standing in the program. Pre- 
service activities include training new mentors; 
training FLIGHT staff in the creation and distribution 
of Student Detail Reports; and creating lesson plans 
for each of the six college access workshops and the 
three supplemental workshops (nine annual 
workshops in total). Student service activities are 
provided directly to students (e.g., wrap around case 
management; one-on-one mentoring; college access 
workshops; supplemental workshops) or to their 
parents and guidance counselors in the form of the 
Student Detail Reports. 


Abt Associates 


Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 


Fidelity Was 
Measured 


Teaching of Curriculum Develop/Institute New Curriculum & 1 NO 

Materials 
n and Higher Education for Tomorrow (FLIGHT) 

Train mentors Provide PD for Other School or Non- 2 YES 
School Staff 

Train FLIGHT staff on Provide PD for Other School or Non- 

application of Student Detail School Staff 2 YES 

Report (SDR) 

Develop lesson plans for Develop/Institute New Curriculum & 2 YES 

workshops Materials 

Case management meetings Provide Services Targeting 2 YES 
Individualized Learning 

Student mentoring Provide Services Targeting 2 YES 
Individualized Learning 

College Access and Success Support College/Career Readiness 2 YES 

Workshops (CASWs) 

Supplemental Student Support College/Career Readiness 2 YES 

Workshops (SWs) 

SDR Plan for and Support Assessment & y) YES 
Data Use 
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Implemented 
Number of Times with 
Fidelity Was Adequate 


Intervention Description 


Florida Master Teacher Initiative (FMTI) 


Key Component Name 


Key Component Domain 


Measured Fidelity? 


The Florida Master Teacher Initiative (FMT!) Early Childhood Education Provide PD for Teachers 3 No 
supports teachers and administrators through a job- Graduate Program 

embedded master’s degree program, a Teacher 

Fellows program, a Principal Fellows program, anda | Teacher Fellows Program Provide PD for Teachers 3 YES 
Summer Leadership Institute. The goal of FMTI is to _, 

foster professional learning communities, teacher Principal Fellows Program eee Other School or Non- 3 NO 
professionalism, and improved classroom practices, petunia 

and ultimately to improve students’ reading and math | Summer Leadership Institutes | Provide PD for Teachers and Other Staff 3 NO 
achievement. 

InnovateNYC Ecosystem 

ee ane yn eit School-App Matching Develop/Institute New Curriculum & 1 NO 
achievement, engagement, and self-efficacy by Materials 

improving the quality of education technology tools 

that teachers use in the classroom, and thereby Teacher/Developer Provide PD for Teachers 1 YES 
increasing teachers’ use of technology. The Engagement 

intervention employs a bottom-up strategy through ‘ ; 

which teachers and other educators are directly Nie of Apps as oo New Curriculum & 1 NO 
engaged in the process of identifying student ae, ee 

learning challenges and selecting the solutions to 

address those challenges. 

Innovations in Early Mathematics 

The overarching goal of the Innovations in Early Learning labs Provide PD for Teachers y) YES 
Mathematics program is to use a whole teacher 

approach to professional development to help Summer institutes Provide PD for Teachers 1 NO 
students in Pre-K to 3rd grade reach or exceed state . : 9 YES 
learning standards in mathematics. The intervention Coaching Te ee ie earls 

includes summer institutes, learning labs, and onsite | Grade level meetings Support Staff Collaboration 2 YES 
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coaching for teachers, as well as leadership 
meetings to address three significant problems 
confronting mathematics education and learning: (1) 
the mathematics achievement gap between low- 
income, minority students and their more 
advantaged peers; (2) the low level of competence 
in math teaching of early childhood and early 
elementary teachers; and (3) the limited and 
ineffective in-service opportunities for early 
childhood and elementary teachers to develop 
mathematics competencies. 


Key Component Name 


Leadership academy 


Integrating English Language Development (ELD) and Science 


Key Component Domain 


Target Leadership Structures and 
Supports 


Implemented 
Number of Times with 


Fidelity Was Adequate 
Measured Fidelity? 


2 YES 


The Integrating English Language Development 
(ELD) and Science intervention is designed to close 
the achievement gap between students with limited 
English proficiency and their peers at the K-5 level 
by supporting adoption of an integrated ELD and 
science instructional program. The intervention 
includes a professional development program 
comprised of workshops, study groups, and 
leadership development activities. The expectation is 
that strategically designed professional development 
focused on integrated ELD and science will increase 
the percentage of highly effective teachers, as 
measured by both teacher performance and student 
achievement in ELD and science. 


Professional development 
(summer workshops and study 


groups) 


Provide PD for Teachers 


2 YES 


Curriculum units 


Develop/Institute New Curriculum & 
Materials 


2 YES 


Teacher and district leadership 


Target Leadership Structures and 
Supports 


2 YES 
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Intervention Description 
Internet-based Reading Apprenticeship Improving Science Education (iRAISE) 


Key Component Name 


Key Component Domain 


Implemented 
Number of Times with 
Fidelity Was Adequate 
Measured Fidelity? 


The year-long Internet-based Reading 


Delivery of PD Content Adheres 


Provide PD for Teachers 


Apprenticeship Improving Science Education ie Standaris 1 YES 
(iRAISE) program seeks to provide completely 
online literacy professional development to Delivery of Professional Provide PD for Teachers 
approximately 100 teachers from 20 schools in Development by Program 1 YES 
Pennsylvania and Michigan. iRAISE starts with a5- | Developers 
day foundational training, followed by monthly Ignite 
Sessions and Professional Learning Communities Teacher Attendance at Provide PD for Teachers 1 NO 
(PLCs) aimed at influencing teacher instruction and Professional Development and 
increasing teacher knowledge of literacy instruction. FLQoie canes 
The goal is to improve students’, especially high- 
needs students’, general reading literacy skills and 
ability to engage in and understand a variety of 
scientific texts. 
Making Time for What Matters Most 
Making Time for What Matiers Most is a schootwide | Fiexihle suudent Scheduling Institute Structural Change 2 YES 
intervention consisting of three major elements: (1) 
provide structures and supports to facilitate student | College Access Time Support College/Career Readiness 2 NO 
mastery of academic material and successful ; ; ; 
Professional Learning Support Staff Collaboration 2 NO 


completion of all core courses in one year or less; 
(2) provide a range of personalized supports to 
students to increase engagement in school and 
promote college readiness; and (3) improve 
teachers’ pedagogical and student support practices 
to maximize the effectiveness of increased learning 
time. The intervention aims to reduce dropout rates, 
increase academic achievement, and increase high 
school graduation rates while also strengthening 
students’ college readiness skills and increasing the 
percentages of students who graduate high school 
and go on to college. 


Communities 
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Intervention Description 
New England Network for Personalization and Performance (NETWORK) 


Key Component Name 


Key Component Domain 


Number of Times 
Fidelity Was 
Measured 


Implemented 
with 
Adequate 
Fidelity? 


The New England Network for Personalization and 
Performance (NETWORK) aims to prepare high 
school students for postsecondary success by 
personalizing learning for all students in participating 
schools. The NETWORK accomplishes this through 
a collaborative, peer-mentored approach to 
professional development that supports the 
development of instructional experiences and related 
assessments and rubrics; a change leadership team 
in each network school that shepherds systemic 
change; a Performance Assessment Review Board 
(PAR), made up of nationally recognized experts 
who make visits to the network schools, that 
assesses student experiences and provides 
feedback for improvement; and a Project Steering 
Committee that oversees the project, including 
monitoring progress toward goals and adherence to 
the timeline and budget. 


Development of inquiry-based 
curricular units and 
performance-based 
assessments, along with rubrics 
aligned with state content 
standards 


Establishment of a broadly 
representative Project Steering 
Committee (PSC) responsible 
for project oversight, 
communication, dissemination, 
and evaluation 


Develop/Institute New Curriculum & 
Materials 


Target Leadership Structures and 
Supports 


YES 


YES 


Establishment of a Performance 
Assessment Review Board 
(PAR) comprised of nationally 
recognized experts 


Target Leadership Structures and 
Supports 


YES 


Establishment of school-based 
change leadership (CL) team 
with the support of a school 
change coach 


Target Leadership Structures and 
Supports 


YES 


Objective evaluation process 
used to provide feedback and 
documentation for the Project 
Steering Committee (PSC) 


Target Leadership Structures and 
Supports 


YES 
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Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 
Fidelity Was 
Measured 


Oakland Accelerates 
Oakland Accelerates is a pean 0 bring the AP Teacher Professional Provide PD for Teachers 2 YES 
College Board's EXCELerator district-wide process Development 
for college readiness to Oakland Unified School 
District. The intervention includes PD for teachers, | Capacity and Policy Guidance | Support College/Career Readiness 2 NO 
leadership, and counselors to support college-going oe . 
culture: support and coaching for College Readiness College Readiness Specialist Provide Coaching for Staff 9 YES 
Specialists; technical assistance to develop policy Coaching and Support 
and infrastructure to support college-going culture; | District Diagnostic Support College/Career Readiness 1 YES 
and key resources to support teachers, students, 
and families to build an understanding of college 
preparation. 
Ounce of Prevention Fund Professional Development Initiative (PDI) 
The primary goal of the Ounce of Prevenvon Funds Center Leader Professional Provide PD for Other School or Non- 2 YES 
Professional Development Initiative is "to build birth- Development School Staff 
to-five teachers' capacity to design and deliver 
standards-aligned, data-driven instruction and to Coach Induction (Year 1) and Provide PD for Other School or Non- 3 YES 
close developmental and learning gaps among high =| Community of Practice School Staff 
needs students to support their kindergarten Implemented by the Sponsor 
readiness through simultaneous job-embedded PD Organization (Year 2 and 3) 
for teachers, leaders, and their coaches. ; ; 
Coach Professional Provide PD for Other School or Non- 2 YES 
Development School Staff 
Direct Supervisor Professional Provide PD for Other School or Non- 2 YES 
Development School Staff 
Professional Development Provide Coaching for Mixed Group y) YES 
Initiative Implementation 
Teacher Professional Provide PD for Teachers 2 YES 
Development 
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Pathways to STEM Initiative (PSI) 


Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 
Fidelity Was 
Measured 


The Pathways to STEM Initative aaa an integrative | Classroom Technology Develop/Institute New Curriculum & 2 YES 
approach to introduce a STEM curriculum, and Katenals 
includes external mentoring, teacher professional 
development, and extra-curricular opportunities for Implementation of Clark County | Develop/Institute New Curriculum & 2 NO 
students. The program begins in middle school School District & Project Lead Materials 
(Level 1—-Discover) when all students are introduced | The Way Curricula 
to STEM through Gateway to Technology (GTT) . . 
: ; 2 NO 
Siikaculunt: Interested! shudenis Gar-cantialle Implementation of STEM Club Support College/Career Readiness 
involvement by exploring STEM-related careers Math & Science Tutoring Provide Services Targeting D YES 
through extra-curricular activities (Level 2-Explore), Individualized Learning 
and pursue STEM coursework and internships in 
STEM Summer Camp Develop/Institute New Curriculum & y) YES 
Materials 
Teacher Professional Provide PD for Teachers 2 YES 
Development 
Weekly Sessions with STEM Support College/Career Readiness 2 NO 
Professionals 
Problem-Based Learning (PBL) 
diScdle Sorrell, iene i @ he ee , Focusing on 1st Generation Develop/Institute New Curriculum & 4 YES 
ie oe, eee ‘ ste an Me a . ) College Bound Students and Materials 
redefine the relationship between the high schoo Developing a PBL Laboratory: 
and the community by connecting students with local Starting Strong 
professionals in STEM fields to provide real-world 
validation for students’ college and career questions | Increasing Rigor and Focusing Develop/Institute New Curriculum & 5 YES 
and by having counselors help every student to on 21st Century Skills in Materials 
complete at least one college application; (2) provide | Curriculum: PBL Course 
increased instructional time for limited English Implementation 
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Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 


Fidelity Was 
Measured 


Abt Associates 


proveleatstieens cal ewe Gisanulies, pil Developing Distributed Target Leadership Structures and 5 YES 
a focus on mathematics and one-to-one mentoring; : : 
. : : Expertise to Support a Rigorous | Supports 
(3) design and implement a scalable, sustainable, Curriculum: Leadership Team 
21s century skills-based program that includes PBL : 
curriculum in both AP and non-AP courses, specific | Development/Adaptation/Adopti | Develop/Institute New Curriculum & 5 YES 
supports for struggling students, and professional on of Research-Based Materials 
development that trains teachers to design and Framework for PBL Design, 
implement effective problem-based curricula. The Implementation and Evaluation 
aim is to increase STEM learning in the classroom in ma. ; 
order to improve students’ college and career Monitoring and Supporting Provide Services Targeting 5 YES 
readiness and increase their performance on state Career Readiness: EPIC Individualized Learning 
math and science, as well as AP, exams. Campus Ready Survey, 
Application Support 
Increasing Teacher Pedagogical | Provide PD for Teachers 5 YES 
Expertise (SILT) 
Designing a Rigorous Support Staff Collaboration 5 YES 
Curriculum: PBL Courses 
through PBL Design Teams 
PTA Comunitario 
ge Pomuniald seeks to establish Ssuccesstul Phase |: Community Organizing | Involve Parents/Community Members 2 YES 
partnerships between community-based and Mobilization 
organizations and schools with minority and low- 
income families in order to increase student Phase II: Initiation and Involve Parents/Community Members 2 YES 
achievement, particularly for students who are low- Relationship Setting 
income, Hispanic, or limited English proficient (LEP). ay 
The project has three main objectives: (1) establish Phase Ill: Taking Collaborative | Involve Parents/Community Members : vies 
PTA Comunitarios in South Texas public school een is Student SUCCES) 
Educational Leadership Projects 
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districts with a minimum of 20 active members per 
PTA Comunitario; (2) establish a partnership 
between PTA Comunitarios and schools through 
training and technical assistance; and (3) facilitate 
the development of at least one educational 
leadership project informed by actionable data for 
each PTA Comunitario and partner school. 


Key Component Name 


Phase IV: Instituting Sustainable 
Connections 


Key Component Domain 


Involve Parents/Community Members 


Implemented 
Number of Times with 


Fidelity Was Adequate 
Measured Fidelity? 


2 YES 


Public School Choice Initiative (PSC) 


The Public School Choice Initiative (PSCI) supports 
the Los Angeles Unified School District's Bold 
Competition, which aims to create a diverse portfolio 
of innovative schools, supported and sustained, that 
can respond to the needs of the local community 
and turnaround chronically low performing schools. 
The intervention identifies “focus schools” (existing 
schools with low performance) and “relief schools” 
(newly constructed schools designed to ease 
overcrowding in low-performing schools), and 
includes elementary, middle, and high schools. 
Schools submit proposals for funding programs, 
which may include STEM academies; bilingual 
immersion programs; arts-focused schools; or the 
Linked Learning model, which provides students a 
college-preparatory academic curriculum, technical 
education, work-based learning opportunities, and 
various socio-economic and academic supports. 


Identification of Public School 
Choice (PSC) schools 


Select/Evaluate Staff 


2 YES 


Facilitation of stakeholder 
involvement 


Involve Parents/Community Members 


2 YES 


Facilitation of support and 
oversight 


Accountability and monitoring 


Provide PD for Other School or Non- 
School Staff 


Select/Evaluate Staff 


2 YES 


2 YES 
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Responding Effectively to Assessments with Curriculum and Teaching (REACT) 


Key Component Name 


Key Component Domain 


Implemented 
Number of Times with 
Fidelity Was Adequate 
Measured Fidelity? 


Responding Effectively to Assessments with 
Curriculum and Teaching (REACT) focuses on a 
district-wide implementation of support structures of 
data coaching, data informed instruction, and data 
professional learning community teams with the goal 
of increasing teacher effectiveness, improving 
student achievement, and closing the achievement 
gap as measured by scores on California state tests. 
As part of REACH, the Del Norte County Unified 
School District implemented data-driven instruction 
in K to 12th grade classrooms using data teams with 
site/district coordinators. 


Administrators 


Target Leadership Structures and 
Supports 


2 YES 


Classroom Teachers 


Plan for and Support Assessment & 
Data Use 


2 YES 


Data Coaches 


Plan for and Support Assessment & 
Data Use 


2 YES 


Rio Grande Valley Center for Teaching and Leading Excellence: New Teacher Training (NTT) 


The Rio Grande Valley Center for Teaching and 
Leading Excellence (RGV) New Teacher Training 
(NTT) programs aim to increase the supply of 
effective teachers by offering new teachers a five- 
day summer institute, ongoing professional 
development, and periodic coaching from an 
instructional coach during their first year. Through 
these services, NTT aims to improve teacher 
outcomes (e.g., job satisfaction, self-efficacy, 
leadership skills, teacher retention) as well as 
student outcomes (i.e., reading, math, social studies, 
and science achievement on the Texas state 
assessments). 


New Teacher Recruitment and 
Selection 


Select/Evaluate Staff 


3 YES 


New Teacher Training 


Provide PD for Teachers 


3 YES 
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Implemented 
Number of Times with 
Fidelity Was Adequate 


Intervention Description 
Rural Math Excellence Partnership (RMEP) 


Key Component Name 


Key Component Domain 


Measured Fidelity? 


The Rural Math Excellence Partnership (RMEP) 
Project aimed to develop a rural workforce qualified 
for science, technology, engineering, and math 
(STEM) jobs in their local communities. The goal of 
the RMEP Project was to develop and implement a 
model of shared responsibility among families, 
teachers, and communities in rural areas to prepare 
students to be successful in advanced high school 
and postsecondary STEM studies. Via a gap 
analysis and development of a Math Advanced 
Study guide; professional development and ongoing 
coaching for participating teachers; Family Math 
Nights conducted by teachers; a project website and 
social media presence; community-based STEM 
events; and access to technology for students, the 
project aimed to improve student achievement in 
math, attitudes toward math, and interest in STEM 
careers; to increase the percentage of 10th grade 
students enrolled in advanced STEM courses; and 
ultimately to ensure that students leave school 
ready, at a minimum, to enroll in a certificate 
program for a technician-level career in STEM- 
related fields. 


Access to technology 


Develop/Institute New Curriculum & 


2 NO 

Materials 
Community-based STEM career | Involve Parents/Community Members 3 YES 
event 
Family math night Involve Parents/Community Members 2 YES 
Identification of foundational Develop/Institute New Curriculum & 2 YES 
math content gaps Materials 
Professional development and Provide PD for Teachers 2 YES 
ongoing coaching 
Project website Institute Structural Change 2 YES 
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Implemented 
Number of Times with 


Fidelity Was Adequate 
Intervention Description Key Component Name Key Component Domain Measured Fidelity? 


School of One 


School of One provides middle school students with | Operational Aspects Institute Structural Change 2 YES 
personalized, effective, and dynamic math |} fd 
instruction that is tailored to their academic needs, Required Resources Develop/Institute New Curriculum & 9 YES 
interests, and learning preferences. Through an Materials 
adaptive technology platform, the intervention - 

: ae Sufficient Time Institute Structural Change 2 YES 


incorporates multiple instructional modalities 
(including live, online, and collaborative instruction), 
which aim to improve math achievement, student 
academic behaviors and attitudes, teacher and 
student technological knowledge, and teacher 
fulfillment. The ultimate goal is to increase student 
learning and preparation for high school, college, 
work, and life. 


Schools to Watch (STW): School Transformation Network Project 


ae enulsta Wale teenie) Mentor school Provide Coaching for Staff 4 YES 
Transformation Network Project is a whole school 

reform model for the middle grades consisting of a Implement early indicators Plan for and Support Assessment & 4 NO 
guiding framework for high performance, a multi- program Data Use 

layered system of support, and school improvement . . . ; 

strategies. The intervention provides a variety of Lalas aa and state Hele oe Structures and 4 YES 
resources including tools for assessment, goal ce ca el 

setting, action planning and monitoring; a STW- Implement professional learning | Support Staff Collaboration 4 YES 
trained coach to provide ongoing assistance; a communities (PLCs) 

principal leadership coach; a mentor school (i.e., a 

and professional development for teachers. The development designed to build a j vies 
goals of the intervention are to build the capacity of | !earning community and 


persistently low-performing middle-grades schools | address the needs of students 
who need additional support 
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Key Component Name 


Key Component Domain 


Implemented 
with 
Adequate 
Fidelity? 


Number of Times 


Fidelity Was 
Measured 


by a eeveenng eiganizauore ieee ude Create a powerful vision for high | Target Leadership Structures and 
and processes for continuous improvement; 2) 4 YES 
; ; Rane : i performance using the STW Supports 
improving academic rigor; 3) promoting equity for all criteria 
students; and 4) developing an array of supports 
designed to meet the developmental needs of Engage in an in-depth Provide Coaching for Mixed Group 
students. The outcomes of the project are to improve | assessment and planning 4 YES 
the educational practices, experiences, and process using the STW criteria 
outcomes of low-performing middle-grades schools. ; 
STW coach Provide Coaching for Mixed Group 4 YES 
Principal mentor Provide Coaching for Administrators 4 NO 
Science, Technology, Engineering and Math Education for the 21st Century (STEM21) 
sence, Technology, Engineering and May Teacher Professional Provide PD for Teachers 3 YES 
Education for the 21st Century (STEM21) builds on 
: ; Development 
the success of the Connecticut Career Choices 
program and aims to improve student achievement, | Grade Level Implementation Develop/Institute New Curriculum & 3 YES 
interest, and self-efficacy in STEM, as well as Materials 
college enrollment. The intervention provides . . . 
professional development to teachers and delivers a Program Level Implementation oes aes New Curriculum & 3 YES 
STEM course sequence in an interactive onan 
environment in which students progress through 
online coursework that is guided by teachers and 
integrated with authentic learning experiences. 
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Implemented 
Number of Times with 
Fidelity Was Adequate 


Intervention Description 
Spheres of Proud Achievement in Reading for Kids (SPARK) Program 


Key Component Name 


Key Component Domain 


Measured Fidelity? 


Spheres of Proud Achievement in Reading for Kids 
(SPARK) program aims to improve reading 
achievement for students in grades K-3 by providing 
in-school tutoring and increasing parental 
engagement. SPARK students are provided one-on- 
one tutoring during the school day for 30 minutes, up 
to three times per week, for two years. At each 
SPARK site, a program manager, who is also a 
certified teacher, oversees the tutors. The tutoring 
component of SPARK is loosely based on the 
Reading Recovery program, which focuses on in- 
school tutoring with lesson plans written, and 
assessments analyzed, by the tutors themselves. To 
execute the family engagement component, each 
site has a parent partner who works with each 
participating student’s family. Their work is designed 
to bridge the divide between school and home by 
translating literacy concepts, educating families 
about a variety of literacy activities, and validating 
the literacy practices already happening in the home. 
Parent partners stay connected with families through 
a monthly newsletter, monthly family events at each 
site, and phone calls or emails. These 
communications are designed to keep families 
aware of student progress in SPARK, help families 
promote literacy at home, and address any 
attendance issues that arise during the program. 
Parent partners also conduct home visits for all 
students twice during the summer between their first 
and second year of participation. 


In-school tutoring 


Provide Services Targeting 
Individualized Learning 


1 YES 


Family engagement 


Involve Parents/Community Members 


1 YES 
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STEM Learning Opportunities Providing Equity (SLOPE) 


Key Component Name 


Key Component Domain 


Implemented 
Number of Times with 
Fidelity Was Adequate 
Measured Fidelity? 


STEM Learning Opportunities Providing Equity 
(SLOPE) aims to improve academic achievement in 
mathematics and increase interest in STEM courses 
through the use of real-world, project-based 
instruction with a focus on STEM topics. The 
intervention consists of professional development 
and coaching for teachers on the use of project- 
based Algebra and pre-Algebra curricula. Rising 
eighth graders who are low performing students are 
eligible to participate in a 4-week summer program 
(Summer College Awareness and Math Proficiency, 
or C.A.M.P.) focused on pre-Algebra concepts. The 
core curriculum is complemented by a college 
awareness curriculum that highlights pathways to 
STEM courses and careers. 


C.A.M.P. Curricular Materials Develop/Institute New Curriculum & 2 YES 
Materials 

C.A.M.P. Professional Provide PD for Teachers 2 YES 

Development 

C.A.M.P. Coaching Provide Coaching for Teachers 2 NO 

C.A.M.P. College Awareness Develop/Institute New Curriculum & 2 NO 

Curriculum Materials 

Algebra | Curricular Materials Develop/Institute New Curriculum & 2 YES 
Materials 

Algebra | Professional Provide PD for Teachers 2 YES 

Development 

Algebra | Coaching Provide Coaching for Teachers 2 YES 

Algebra | College Awareness Develop/Institute New Curriculum & 2 NO 

Curriculum Materials 

C.A.M.P. Teacher Develop/Institute New Curriculum & 

Implementation of Classroom _| Materials 2 NO 

Intervention, Unit 1 

C.A.M.P. Teacher Develop/Institute New Curriculum & 

Implementation of Classroom Materials 2 NO 

Intervention, Unit 2 

C.A.M.P. Teacher Develop/Institute New Curriculum & 

Implementation of Classroom _| Materials 2 NO 

Intervention, Unit 3 
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Key Component Name 


Key Component Domain 


Implemented 
Number of Times with 
Fidelity Was Adequate 
Measured Fidelity? 


Algebra | Teacher Develop/Institute New Curriculum & 
Implementation of Classroom Materials 2 YES 
Intervention, Unit 1 
Algebra | Teacher Develop/Institute New Curriculum & 
Implementation of Classroom _| Materials 2 YES 
Intervention, Unit 2 
Algebra | Teacher Develop/Institute New Curriculum & 
Implementation of Classroom _| Materials 2 NO 
Intervention, Unit 3 
STEM Summer Learning with VEX Robotics 
The Baltimore STEM Summer Learning Program Instruction in mathematics Develop/Institute New Curriculum & 3 YES 
with VEX Robotics aims to increase mathematics Materials 
knowledge and interest in STEM careers and college 
majors by providing professional development in Professional development in Provide PD for Teachers 3 NO 
mathematics and robotics to teachers, as well as 5 robotics 
weeks of summer instruction in these subjects to j . ; 
students. Specifically, the project provides Professional development in Provide PD for Teachers 2 NO 
professional development to participating teachers to matt 
equip them to lead students in construction of a Instruction in robotics Develop/Institute New Curriculum & 3 YES 
robot and activities to prepare students for a robotics Materials 
competition, as well as high quality delivery of math 
instruction. The robotics component is expected to 
increase student engagement (including attendance) 
and perception of the relevance of mathematics, 
leading to increased student effort and math 
achievement. 
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Implemented 
Number of Times with 


Fidelity Was Adequate 
Intervention Description Key Component Name Key Component Domain Measured Fidelity? 


Texas Tech University “Tech Teach” Program 


The Texas Tech University “Tech Teach” Program is 
a competency-based teacher and school 
intervention. Part A focuses on competency-based 
strategies at the teacher candidate and program Student Teaching Using Provide PD for Teachers 2 YES 
levels via three main components: (1) trainee Competency-Based Framework 
screening and recruitment; (2) competency-based 
pre-certification coursework; and (3) student 
teaching using competency-based training. Part B 
focuses on implementing a model of technology- 
enabled competency-based teacher development in 
mathematics via three main components: (1) 
individual pre-and post-conferences; (2) a 
professional development cycle; and (3) Texas 
Instruments MathForward curriculum. Both parts 
include professional development with coaching, 
multiple assessments of teaching, and professional 
learning communities. 


Transforming Teacher Talent (t3) System 


Aspire Public Schools aims to increase student 
achievement in grades 3-11 through its 
Transforming Teacher Talent (t3) system. This 
intervention uses a train-the-trainer model in which 
experienced and highly effective Aspire teachers PD content library Provide PD for Teachers 1 NO 


Competency-Based Pre- Provide PD for Teachers 2 YES 
Certification Coursework 


Trainee Recruitment and Select/Evaluate Staff 2 YES 
Screening 


Charter Management Provide PD for Teachers 
Organization provides adequate 1 YES 
support 
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Intervention Description 

receive training on the use of three t3 tools and then 
provide professional development and coaching to 
other Aspire teachers. Tools include a professional 
development content library, peer observations 
cycles, and a virtual professional learning 
community. Under its i3 grant, Aspire implemented 
t3 in all 35 Aspire schools in California. 


Key Component Name 


Peer observation/walkthrough in 
BloomBoard (an online 
personalized/customized 
observation data collection tool 
through which principals and 
peer observers conduct informal 
observations and provide 
targeted, frequent feedback and 
exposure to new teaching 
strategies). 


Key Component Domain 


Provide Coaching for Teachers 


Implemented 
Number of Times with 


Fidelity Was 
Measured 


Adequate 
Fidelity? 


Virtual collaborations Support Staff Collaboration 1 NO 
Turnaround with Increased Learning Time (TILT) 
The Tumaround with Increased Reatning: Time More Time for Academics Institute Structural Change 3 NO 
(TILT) model was implemented in two low- 
performing Boston middle schools. Both of the TILT | More Time for Enrichment Institute Structural Change 3 NO 
schools redesigned their school days to include 300 Planning C. ; fsenaie-Sinetaial Gh 3 NO 
more hours per year of additional instructional time ae eee 
for all students. The increased time was allocated to Sustainability Component Institute Structural Change 1 NO 
meeting three goals: more time for core academic 

Teacher Collaboration Support Staff Collaboration 3 NO 


instruction that is data-based and targeted to meet 
the individual needs of students; more time for 
robust enrichment program supports by strong local 
partnerships; and more time for teachers to 
collaborate to improve their effectiveness and 
instructional quality. 


Abt Associates 
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Implemented 


Number of Times with 
Fidelity Was Adequate 
Intervention Description Key Component Name Key Component Domain Measured Fidelity? 
We Are A Village 
We Ne A Village focuses iia sane the school "Parents are Power!" resource Involve Parents/Community Members 2 YES 
readiness and academic achievement of young carilers 
children, from preschool to third grade, by integrating 
parent and family engagement strategies into Cross-system educational Involve Parents/Community Members 2 YES 
classrooms. Project activities are designed to workshops for parents 
support the successful transition of children from 7 
preschool to elementary school by providing Cross-system training for staff & | Provide PD for Teachers and Other Staff 9 NO 
opportunities for cross-system, cross-grade training | Parent Peer Navigators 
for parents, families, and staff; parent and family Direct parent outreach Involve Parents/Community Members 2 NO 
school leadership opportunities; peer support school 
transition groups; and the Incredible Years (IY) Incredible Year (IY) Program Provide PD for Teachers and Other Staff 2 YES 
Teacher Classroom Management and BASIC Kindergarten transition support | Involve Parents/Community Members y) YES 
Parenting Programs. By conducting targeted for families 
outreach to parents and families, including parents 
and families of children with disabilities and/or Parent leadership Involve Parents/Community Members 2 YES 
limited English proficiency, the project encourages 
parents and families to become actively engaged in 
school activities. 
Write Up! 
bass pee mie Hned een We Be ar high Teacher participation in Provide PD for Teachers 3 NO 
quality assessments and standards and improve trainings 
teacher effectiveness through a variety of research- 
based program strategies including support of Step | Teacher collaboration Support Staff Collaboration 3 YES 
Up to Writing; launch of My Access!, an artificial . 3 YES 
intelligence scoring engine; and provision of an Classroom observations Select/Evaluate Staff 
online writing program (i.e., United Streaming by Coach/Teacher on Special Provide PD for Teachers and Other Staff 
Discovery Education). Write Up! also includes Assignment (TSA) participation 3 YES 
professional development to create and integrate in trainings 
online courses and staff development focused on 
improving student achievement through the use of Online lessons and courses in Develop/Institute New Curriculum & 3 NO 
writing instruction Materials 


Abt Associates 
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Implemented 
Number of Times with 


Fidelity Was Adequate 
Intervention Description Key Component Name Key Component Domain Measured Fidelity? 
. : 
eehnoreay. me ere ee Online lessons and courses in Support College/Career Readiness 
teacher effectiveness; upper elementary, middle, 


and high school student writing and English eaieet alld co See Ueaaliees 


language arts achievement; and secondary students’ | Student use of writing software | Develop/Institute New Curriculum & 3 NO 
college and career knowledge. Materials 
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Appendix B: i3 Review Protocol, Version 1.0 


The i3 Review Protocol Version 1.0 (April 7, 2016) was developed to apply the What Works 
Clearinghouse™ (WWC) Standards to reviews of the impact studies from i3-funded evaluations. The i3 
Review Protocol defines the scope and purpose of the review of i3-funded evaluations relative to the 
WWC Standards and defines all topic-specific applications of the WWC Standards for the review of i3 
evaluations. The reviews use a single protocol both to ensure consistency across reviews and to include 
evaluations whose topics fall beyond the scope of existing WWC review protocols. This protocol is 
designed to be as consistent with WWC review protocols as possible using a parsimonious set of rules 
that can be applied across topics covered by i3-funded evaluations. 


Purpose Statement 


The purpose of this Review Protocol is to describe how the Abt Team applies the WWC Standards in the 
review of the i3-funded evaluations. Reviews conducted by the Abt Team assign unofficial WWC ratings 
to i3 impact evaluations. The ratings are unofficial because they are not assigned by the WWC and will 
not be included in the WWC database of reviewed studies.’ In addition, while the WWC reviews only 
publicly available reports, the Abt Team collects the information necessary for the reviews directly from 
the evaluators and includes in the i3 reviews only the prespecified impact analyses for each evaluation. 


The reviews assign one of four ratings to each estimated impact reported to the Abt Team: 


e Unofficially Meets WWC Standards without Reservations. 

e Unofficially Meets WWC Standards with Reservations. 

e Unofficially Does Not Meet WWC Standards. 

e Unofficially Ineligible for WWC Review 

Eligibility Criteria 

Eligible Interventions 

All i3-funded interventions are eligible for review under this protocol. 

Eligible Research 

All evaluations supported by i3 funding are eligible for review under this protocol. 


Eligible Designs 


Randomized controlled trials (RCT) and quasi-experimental designs (QED) are eligible for review under 
this protocol. Regression discontinuity designs (RDD) and single-case designs (SCD) are not eligible for 
review under this protocol but will be reviewed by the WWC. All other designs are classified as ineligible 
for review under this protocol. 


°! The WWC will conduct official reviews of publicly available i3 evaluation reports as they are released. Once 


completed, the reviews can be found at https://ies.ed.gov/ncee/wwc/ReviewedStudies. 
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Eligible Analyses 


Analyses that estimate the effect of an educational intervention relative to a comparison condition on an 
outcome are eligible for review under this protocol. The comparison could be to business-as-usual or to 
an alternative intervention. 


Eligible Outcomes 


All outcomes are eligible for review under this protocol. These outcomes may include student outcomes 
(including, but not restricted to, individual or aggregate measures of student achievement) as well as 
outcomes for teachers and other parties that could be affected by the i3 interventions. 


Eligible Subgroups 


Subgroup analyses are eligible for review; the reviews are not limited to particular subgroups. 


Evidence Criteria 


Estimated impacts from the impact study that meet the eligibility criteria listed above are assessed against 
the standards described in the WWC Procedures and Standards Handbook, version 3.0, Section III: 
Screening and Reviewing Studies (pp. 7-21). This section describes the application of these standards in 
the review of i3-funded evaluations. 


Level of Causal Inference 


For evaluations that assign clusters of individuals to groups (e.g., treatment and control), the evidence 
criteria for sample attrition and baseline equivalence described below account for the level at which 
inferences are made from the impact study results. 


Causal inference at the cluster level: Inference from estimated impacts about an intervention’s effect on 
clusters of students (e.g., clusters of students who are in the same classrooms, schools, or districts at the 
point the outcome is measured). 


Causal inference at the individual level: Inference from estimated impacts about an intervention’s effect 
on individuals (e.g., individual students or teachers). 


Sample Attrition 


The reviewers apply the WWC’s liberal attrition standard unless there is reason to believe that attrition 
will yield substantial bias in the estimated effect—more specifically, when attrition is endogenous to the 
intervention and data are not missing at random. The reviewers assume that attrition will yield substantial 
bias in the estimated effect and therefore apply the conservative attrition standard if all three of the 
following conditions hold: 


1. The sample consists exclusively of high school students. High school students typically have more 
choice about whether to remain in school, change schools, or change classes: these decisions are not 
likely to be random. 


2. Mobility decisions affect attrition—that is, whether a student is included in the analytic sample. 
For example, this condition would be satisfied if the i3 local evaluator collects outcome data on 
outcomes only from the schools initially included in the impact study sample so that no outcome data 
are collected on students who moved out of these schools. 
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3. Mobility decisions almost surely affect the value of the outcome variable. The reviewers treat the 
following outcomes as almost surely affected by the mobility of high school students across schools 
or into and out of school (i.e., dropping in and dropping out): graduation, credit completion, 
promotion to the next grade level, dropout. The reviewer reserves the right to designate other related 
variables as being almost surely affected by student mobility. 


Under these conditions, the reviewer applies the conservative attrition standards because an intervention’s 
effect on student mobility could produce non-random differential attrition and biased impact estimates.” 


To calculate attrition rates in RCTs, the reviewer divides the number of units in the analytic sample by the 
number of units that were in the sample at the point of random assignment.” For RCTs that randomize 
individuals, attrition is deemed acceptable if the combination of overall and differential attrition for the 
individuals that were randomized satisfies the selected standards (liberal or conservative). For RCTs that 
randomize clusters of individuals, attrition is deemed acceptable: 


e for causal inference at the individual level if the analytic sample excludes joiners™ and the 
combination of overall and differential attrition satisfies the selected standards for both the 
individuals and the clusters that were randomized. 


e for causal inference at the cluster level if the analytic sample includes joiners and the combination of 
overall and differential attrition satisfies the selected standards for the clusters that were randomized. 


For more information about the attrition standards, see the WWC Procedures and Standards Handbook 
version 3.0, Section III: Subsection B.2, “Sample Attrition: Is the combination of overall and differential 
attrition high?” (pp. 11-15). 


Baseline Equivalence 


If the impact study design is an RCT with high attrition or a QED, the reviewer assesses whether the 
treatment and comparison groups in the analytic sample are equivalent at baseline. For comparison group 
designs in which assignment occurs at the cluster level (e.g., QEDs that compare student outcomes in 
schools that received the intervention to student outcomes in schools that did not receive the intervention), 
baseline equivalence can be established: 


e for causal inference at the individual level if baseline equivalence is established on the analytic 
sample of individuals prior to the intervention. 


°° For example, suppose an intervention reduces the dropout rate, and students who drop out are excluded from the 


analytic sample. The effects could lead to a lower attrition rate in the treatment group than in the control group, 
lower attrition bias in the treatment group mean than in the control group mean, and biased estimates of the 
treatment effect for outcomes like graduation, grade promotion, and credit accumulation. 


°° The reviewer defines the point of random assignment as the point at which the results of random assignment 


(treatment or control) are communicated to parties that could be influenced by the results (e.g., schools if the 
impact study randomized schools or teachers). 


° Joiners are defined as individuals who are not in the sample before the point of random assignment but are in 


the sample at the point of outcome measurement—that is, they were included in the analytic sample. 
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for causal inference at the cluster level if baseline equivalence is established on the analytic sample 
of clusters prior to the intervention (i.e., on the same cohort of individuals from an earlier point in 
time or an earlier, adjacent cohort in the same grade as the cohort used in the impact analysis”). 


To meet the baseline equivalence criteria, the impact study must both affirmatively demonstrate baseline 
equivalence between the two groups and not provide evidence of nonequivalence. To affirmatively 
demonstrate baseline equivalence, the impact study must show that the two groups are equivalent at 
baseline on a measure that is highly correlated with the outcome measure. 


Two types of pre-intervention measures are always considered highly correlated with a given outcome 
measure: 


1. 


A pre-intervention measure of the outcome measure. To satisfy this condition, the baseline 
measure must be collected using the same testing instrument or protocol as the outcome measure 
(e.g., the Stanford 10 at both time points). 


A pre-intervention measure in the same evaluator-defined outcome domain or, where 
applicable, the same domain as defined for i3. The outcome domains as defined for i3 are 
described in Tables 1 and 2. Briefly, there are seven student outcome domains consistent with the 
specific goals of i3. These “relevant outcomes” are: English language arts, mathematics, science, 
social studies, general achievement, academic performance in school, and educational progress and 
attainment. Additionally, there are also three “other” student outcome domains: student achievement 
in other subjects (beyond the four core academic subjects), labor market outcomes and social- 
emotional skills/development. For example, if the evaluator separates reading outcomes into the 
fluency and comprehension outcome domains, the reviewer will accept a fluency baseline measure 
for a comprehension outcome because both fall into the same outcome domain of English language 
arts defined for i3. 


Additionally, for causal inference at the individual level, the reviewer considers additional pre- 
intervention measures to be highly correlated with the outcome measure in specific contexts and for 
particular outcome measures.” 


If the outcome does not have a natural pre-test (e.g., graduation) and the intervention is provided to 
students at the K-12 or postsecondary level, the following combination of measures are considered 
highly correlated with the outcome: a) race/ethnicity or a pre-intervention measure of disadvantage 
and b) a pre-intervention measure of academic performance or achievement. Acceptable measures of 
disadvantage include eligibility for free or reduced price lunch, immigrant status, parent’s education, 


65 


66 


The earlier adjacent cohort must be from the year that immediately precedes the implementation of the 
intervention—that is, the most recent cohort of students that did not receive the intervention in either of the two 
groups. 


For causal inference at the individual level, the additional options for acceptable baseline measures account for 
the challenges in obtaining a pretest measure in the same domain as the outcome. However, for causal inference 
at the cluster level, these challenges are not present because it is usually possible to establish baseline 
equivalence at the cluster level for a pre-intervention measure in the same domain as the outcome (e.g., cluster 
level outcomes at the same grade level as the intervention but in the prior year). Therefore, additional options 
for establishing baseline equivalence for causal inference at the cluster level are not necessary. 
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single-parent household, special education, and teen parent status. Acceptable measures of academic 
performance or achievement include standardized test scores, credits earned, at or behind grade level, 
rate of school attendance, and grade point average. 


e If the outcome is a measure of student achievement in an outcome domain that is not tested in every 
grade level (e.g., science or social studies), and the intervention is provided to students at the K-12 
level, either of the following two types of measures are considered highly correlated with the 
outcome: (1) a pre-intervention measure in the mathematics outcome domain defined for i3 or (2) a 
pre-intervention measure in the English language arts outcome domain defined for i3. 


e If the outcome is a measure of children’s reading and literacy or mathematics and the intervention is 
provided to children below the age of 3 years 0 months, either of the following two types of measures 
are considered highly correlated with the outcome: (1) a pre-intervention measure in the general 
achievement outcome domain defined for i3 or (2) a pre-intervention measure in the English 
Language Arts outcome domain defined for i3. 


e If the intervention is provided to children younger than 18 months, a pre-intervention measure of 
parental resources (e.g., maternal education, family income, and single-parent household status) or a 
standardized test of maternal intelligence (e.g., Kaufman Brief Intelligence test) is considered highly 
correlated with child outcomes. 


Even if the two groups are equivalent on a pre-intervention measure that is highly correlated with the 
outcome measure, baseline equivalence will be rejected if the evaluator provides evidence of 
nonequivalence for any measure in the same evaluator-defined domain or in the same domain as defined 
for i3 as the outcome measure. Baseline equivalence will also be rejected under either of the following 
two scenarios: 


1. The analysis plan indicates that the study is targeted toward a special population (e.g., English 
learners, students with behavioral problems, or students in special education), evidence from the 
baseline or analytic samples suggest that over 50 percent of the sample belongs to the special 
population, and the evaluator provides evidence of nonequivalence between the two groups in the 
analytic sample on either (1) the fraction of students that belong to the special population (e.g., has a 
behavioral problem) or (2) the mean of a continuous measure of the condition by which the special 
population is defined (e.g., scale scores from an assessment of behavioral problems). 


2. The intervention is provided to children younger than 18 months, and the evaluator provides evidence 
of nonequivalence between the two groups on any measure of developmental delay or disability (e.g., 
a developmental screening instrument of cognitive development or social-emotional development). 
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Exhibit B.1: Relevant Student Outcome Domains Defined for i3 


English language arts. Outcomes related to English language arts measure the development of 
verbal, written, and other forms of communication skills or content knowledge. English language arts 
includes, but is not limited to, alphabetics, phonological processing, reading fluency, English 
language development, writing, print knowledge, oral language, comprehension (including reading 
comprehension, vocabulary, and listening comprehension), spelling, knowledge of features of the 
English language (e.g., grammar, syntax), general reading achievement and general literacy 
achievement. 


Mathematics. Outcomes related to mathematics measure mathematics content knowledge and 
skills, which demonstrate understanding of and application of mathematical concepts, procedures, 
and problem solving. Mathematics content includes, but is not limited to, Knowledge of numbers, 
arithmetic, measurement, graphing, logical reasoning, pre-algebra, algebra, geometry, trigonometry, 
pre-calculus, calculus, and general mathematics achievement. 

Science. Outcomes related to science measure mastery of science concepts, inquiry, practices, 
procedures, and problem solving, such as reasoning and proof, making connections, and use of 
scientific representation. Science content includes, but is not limited to, life science (e.g., biology), 
Earth/space science (e.g., geology, astronomy), physical science (e.g., chemistry, physics), and 
general science achievement. 

Social studies. Outcomes related to social studies measure content knowledge of relationships and 
the functioning of society as well as content application. Social studies content includes, but is not 
limited to, civics, government, geography, history, economics, and general social studies 
achievement. 


General achievement. Either (A) a combination of two or more of the above domains related to 
academic achievement in a given subject area, (B) general academic achievement that is not 
content-specific (including standardized achievement tests like the ACT and SAT as well as state- 
mandated tests that are not content-specific), or (C) cognition, which includes, but is not limited to, 
memory, problem-solving, cognitive processing and flexibility, general Knowledge, and IQ. Measures 
of general kindergarten readiness are considered cognition measures. 

Academic performance in school. Outcomes related to the extent to which students adequately 
complete expected coursework. As such, eligible measures of academic performance in schools are 
those that arise naturally from student educational experiences. Examples of ways that academic 
performance in school might be operationally defined in studies include: (a) final grade in a single 
course, (b) grade point average across courses, and (c) the ratio of courses passed vs. failed. 
Educational progress and attainment. Outcomes related to staying in school, progressing in 
schools, completing school, and postsecondary access and enrollment (from WWC'’s Transition to 
College protocol). (A) Staying in school refers to outcomes that measure whether the student has 
dropped out of school and the number of days the student was enrolled in school. (B) Progressing in 
school refers to outcomes that assess the number of high school course credits the student has 
earned, whether the student was promoted to the next grade, and the highest grade the student has 
completed. (C) Completing school refers to outcomes that measure whether the student has earned 
a high school diploma or GED or whether he or she has graduated from high school. (D) 
Postsecondary access and enrollment refers to the process of applying to, actually enrolling, and 
attending a postsecondary institution. Examples of ways that enrollment might be operationally 
defined in studies include: (a) actual enrollment in college, (b) number and/or selectivity of admitted 
and/or enrolling institutions, (c) enrollment by institution type (2 year vs. 4 year), (d) intensity of 
enrollment (full time vs. part time), (e) timing of enrollment (e.g., immediate vs. delayed enrollment 
after high school). (f) the completion of financial aid forms geared at college attendance, and (g) 
enrollment in college preparatory courses (e.g., Advanced Placement, dual enrollment, or 
International Baccalaureate). 
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Exhibit B.2: Other Student Outcome Domains Defined for i3 


e Achievement in Other (Non-Core Academic) Subjects. Outcomes related to content knowledge 
and skills outside of the four core academic subjects (Mathematics, English Language Arts, 
Science, and Social Studies). Other subjects include, but are not limited to arts, drama, home 
economics, ancient or modern languages, music, physical development or education, and 
technology. 

e Labor Market Outcomes. Outcomes related to measuring student employment status (e.g., 
employed vs. unemployed; employed full-time vs. employed part-time; employed in field vs. not 
employed in field); includes outcomes related to vocational skill development (e.g., training ina 
trade via collaborative education or career academies). 

e Social-emotional skills/development. Outcomes related to social-emotional development 
measure behavioral, social, and emotional competencies. Social-emotional development includes, 
but is not limited to, measures of pro-social (or problem) behaviors, social interactions, cooperation, 
self-concept, engagement, attention, persistence, impulsivity, self-control, and initiative. 


In general, to compute the baseline difference between the two groups, the reviewer uses the unweighted 
mean and standard deviation for the treatment group and the unweighted mean and standard deviation for 
the comparison group. However, when studies use weights or other analytic methods to account for the 
research design or to address nonresponse, the reviewer uses the same method to test for baseline 
equivalence. For example, if an evaluator of an RCT with high attrition uses weights for the impact 
analysis to account for differential rates of assignment to treatment within blocks, the reviewer uses 
weighted means and standard deviations in its assessment of baseline equivalence. If the study takes a 
modeling approach to account for design features of the study (e.g., the model includes fixed effects for 
the blocks within which schools were randomly assigned or selected and random effects for clusters of 
students selected into the study sample), the reviewer uses the following two pieces of information to 
assess baseline equivalence: (1) the unweighted standard deviations for both groups and (2) a “design- 
adjusted” difference in means between the two groups (i.e., the regression coefficient on the treatment 
indicator from a model that regresses the baseline pre-test variable on the treatment indicator and 
whatever fixed and/or random effects were included in the analysis model to account for study design 
features). 


Lastly, the reviewer reserves the discretion to conclude that the study cannot establish baseline 
equivalence if the intervention and comparison groups were drawn from substantially different contexts, 
as reported by the evaluator (e.g., urban schools versus rural schools, schools in corrective action versus 
schools that consistently make Adequate Yearly Progress). 


For more details on the WWC Standards for baseline equivalence, see the WWC Procedures and 
Standards Handbook, version 3.0, Section III: Subsection B.3, “Baseline equivalence: Is equivalence 
established at baseline for the groups in the analytic sample?” (pp. 15-16). 


Outcomes 


The reviewer assesses outcomes for face validity, reliability, consistency of data collection, and (lack of) 
overalignment. To satisfy the outcome standards for face validity, an outcome must be clearly defined, 
have a direct interpretation, and measure the outcome domain that it was intended to measure. To be 
clearly defined, the evaluator must provide as much detail about the outcome as might typically be 
expected in a journal article to provide readers with a basic understanding of how it was constructed. To 
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have a direct interpretation and measure the outcome domain it was designed to measure, the outcome 
must appear to measure the domain into which it was classified by the evaluator. 


The reviewer treats standard educational measures (e.g., grades, credits earned, grade promotion or 
retention, graduation or dropout, and postsecondary attendance, persistence or completion) as face valid 
for the domains they most clearly measure. For example, grades are considered to be face valid measures 
of academic performance in school (but not of student achievement), and both credits earned and 
promotion to the next grade level are considered face valid measures of educational progress and 
attainment. 


The minimum standards for the reliability of the outcome measure are: 


e Internal consistency (such as Cronbach’s alpha) of 0.60 or higher. 
e Temporal stability/test-retest reliability of 0.40 or higher. 


e Inter-rater reliability (such as percentage agreement, correlation, or kappa) of 0.50 or higher. 


Each outcome measure must satisfy one of these three criteria to meet the outcome standards. The 
reviewer accepts both evaluator-reported reliability statistics and references cited in the literature, 
assumes that standardized tests (e.g., the Terra Nova or Stanford 10) and state-required tests satisfy the 
reliability criteria, and exempts standard educational measures from reliability criteria since reliability 
cannot be estimated for these measures. 


In addition, to meet the outcome standards, outcomes must be collected in the same manner between 
treatment and comparison groups and not be overaligned with the intervention. 


For more details on the outcome standards, see the WWC Procedures and Standards Handbook, version 
3.0, Section III: Subsection B.4, “Outcome Eligibility and Reporting” (pp. 16-19). 


Confounding Factors 


The reviewer assesses whether the design used to produce a particular estimated impact suffers from a 
fundamental confound. For information on the criteria for identifying confounds, see the WWC 
Procedures and Standards Handbook, Version 3.0, Section III: Subsection B.5, “Confounding Factors” 
(pp. 19-20). 


In principal, a design that estimates the effects of a bundle of interventions could be viewed as either free 
of confounds for the intervention bundle or confounded for each particular intervention in the bundle. 


e If the evaluation tests a bundle of i3-funded interventions, the reviewer treats the estimated impacts as 
unconfounded for the effects of the bundle (i.e., the design does not suffer from a confound). 


e If the evaluation tests a bundle that includes an i3-funded intervention and a non-i3-funded 
intervention, the reviewer treats the estimated impacts as: 


— unconfounded if the non-i3-funded intervention is offered by the developer of the i3-funded 
intervention (i.e., the design does not suffer from a confound). 


— confounded if the non-i3-funded intervention is offered by a different entity from the developer of 
the i3-funded intervention (i.e., the design suffers from a confound). 
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Statistical Adjustments 


The reviewer makes clustering corrections when testing for statistical significance for analyses that have 
not adequately accounted for clustering (see the WWC Procedures and Standards Handbook, version 3.0, 
Section IV: Subsection B, “Statistical Significance of Findings,” pp. 24—26). Like the WWC, the reviewer 
uses the Benjamini-Hochberg procedure to correct for multiple comparisons. 


Correcting for Multiple Comparisons in Reporting Evidence of Effectiveness from i3 Evaluations 


Most of the i3 local evaluations plan to conduct more than one hypothesis test to determine whether the 
intervention was effective. Conducting multiple hypothesis tests can lead to a high “false discovery rate,” 
which is the proportion of estimated effects that are large enough to reject the null hypothesis when the 
true impact is zero. In addition, it can lead to a high “family-wise error rate,” which refers to the fraction 
of random samples for which evaluators would falsely conclude, purely by chance, that the intervention 
has a positive effect on outcome variables in the same domain, which is defined as follows: 


e Outcome domain. The outcome domain indicates, at a general level, the outcome that may be 
affected by the intervention; it can be thought of as a latent construct that can be measured with one 
or More outcome measures. For the purpose of correcting for multiple comparisons, the reviewer uses 
the relevant student outcome domains defined for i3 (see Table 1). 


The reviewer makes adjustments for multiple comparisons that address outcomes in the same domain. To 
correct study estimated impacts from prespecified analyses for multiple comparisons, the reviewer uses 
the Benjamini-Hochberg procedure as described in the WWC Procedures and Standards Handbook, 
version 3.0, Section IV (pp. 25-26). The reviewer follows the recommendations of an IES methods report 
and does not adjust for multiple comparisons in analyses that were not prespecified.” 


°”  Schochet, P. Z. (2008). Technical methods report: Guidelines for multiple testing in impact evaluations (NCEE 


2008-4018). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, U.S. Department of Education. 
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Appendix C: Assessments of and Findings from 67 13 Impact and 
implementation Evaluations 


Intervention 
Scale-up 
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Strength of the Impact 


Evaluation 


Unofficially Meets WWC 


Quality implementation 


evaluation 


adequately representative of 


Impact evaluation sample 
population served 


Impact evaluation is considered 


independent 


Evidence that the intervention 
was implemented with 


adequate fidelity 


Evidence of at least one 
positive, statistically significant 
impact on student outcomes 


Abt Associates 


KIPP Yes@ Standards with Yes No Yes Yes Yes 
Reservations 
Unofficially Meets WWC 
Reading Recovery Yes? Standards without Yes Yes Yes Yes Yes 
Reservations 
Unofficially Meets WWC No 
Success for All (SFA) Yes Standards without Yes No Yes Yes (null) 
Reservations 
Unofficially Meets WWC No 
Teach for America (TFA) Yes Standards without Yes No Yes Yes (nul) 
Reservations 
aNUALIO 
Advanced ASSET Unofficially Meets WWC No 
i Yes Standards with Yes Yes Yes Yes 
Professional Development (null) 
Reservations 
; : ae No evidence 
Eiuciorescopnan | | OWmptgcname | Ye | Yes | vee | Yes | uot 
9 DNM WWC) 
Beg mgt tk vgs Unofficially Meets WWC 
Sn eEHS eeaes wilaive Yes Standards without Yes Yes Yes Yes Yes 
(CLI) Program 
Reservations 
Collaborative Strategic Enomealy Meet SWE No 
¥ Yes Standards without Yes Yes Yes Yes 
Reading (CSR) s (null) 
Reservations 
Unofficially Meets WWC 
Se Rea roda Yes Standards with Yes Yes Yes No Yes 
(CRP) f 
Reservations 
College-Ready Writers pooncaly Meet ae 
Yes Standards without Yes Yes Yes Yes Yes 
Program (CRWP) : 
Reservations 
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Intervention 
Unofficially Meets WWC No 
Diplomas Now Yes Standards without Yes Yes Yes Yes (null) 
Reservations 
eee unica Meets VC 
; A Yes Standards without Yes Yes Yes Yes Yes 
Teaching Strategies Rescwanan 
(eMINTS) 
GO College (An Enhanced vnemeeN Meet we No 
: Yes Standards without Yes Yes Yes Yes 
Version of Talent Search) ; (null) 
Reservations 
Leadership Assistance for Unofficially Meets WWC No 
Science Education Reform Yes Standards without Yes No Yes Yes (null) 
(LASER) Reservations 
Northeast Tennessee Unofficially Meets WWC 
College and Career Ready Yes Standards with Yes Yes Yes No Yes 
Consortium Reservations 
: eee Unofficially Meets WWC 
Ree A CES Yes Standards without Yes Yes Yes Yes Ne 
(RA) (null) 
Reservations 
Reading Enhances Unofficially Meets WWC No 
Achievement During the Yes Standards without Yes Yes Yes Yes (null) 
Summer (READS) Reservations 
Unofficially Meets WWC 
Slane sete Yes Standards without Yes Yes Yes Yes Yes 
Program : 
Reservations 
Virginia Initiative for Science Unofficially Meets WWC No 
Teaching and Achievement Yes Standards with Yes Yes Yes Yes (null) 
(VISTA) Reservations 
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impact analyses 


Strength of the Impact 


Evaluation 


Unofficially Meets WWC 


Quality implementation 


evaluation 


adequately representative of 


Impact evaluation sample 
population served 


Impact evaluation is considered 


independent 


Evidence that the intervention 


was implemented with 
adequate fidelity 


Evidence of at least one 
positive, statistically significant 
impact on student outcomes 


The Sevevement Newark Yes Standards with Yes Yes Yes No Ne 
(ANet) (null) 
Reservations 
Unofficially Meets WWC 
auvaliess peeoneut nr) Yes Standards with Yes Yes Yes Yes Ne 
Insight (null) 
Reservations 
Advancement via Individual . Unofficially Does Not Meet Ne eigenee 
Determination (AVID) ics WWC Standards nes ics tgs ies Unoticaly 
DNM WWC) 
Unofficially Meets WWC No 
Around the Corner (ATC) Yes Standards without Yes Yes Yes Yes (null) 
Reservations 
Arts Achieve: Impacting phoma Meets Ne No 
; Yes Standards with Yes Yes Yes No 
Student Success in the Arts : (null) 
Reservations 
: Unofficially Meets WWC 
ans jor Leamnlng (2) Yesd Standards without Yes Yes Yes Yes Ne 
Lessons (null) 
Reservations 
Bay State Reading Institute unencaly Meee me No 
Yes Standards with Yes Yes Yes Yes 
(BSRI) : (null) 
Reservations 
Boston Teacher Residency Cnomcaly eels wae No 
Yes Standards with Yes Yes Yes Yes 
(BTR) (null) 
Reservations 
Building Assets, Reducing unofeely Meets We 
A Yes Standards without Yes No Yes Yes Yes 
Risks (BARR) Model ; 
Reservations 
Collaborative Organizational cenuencs 
Model to Promote Aligned Unofficially Does Not Meet 2 
Yes Yes No Yes Yes (Unofficially 
Support Structures WWC Standards DNM WWC) 
(COMPASS) 
_ No evidence 
CollegeYES Yes ae eee Yes | No Yes Yes | (Unofficially 
DNM WWC) 
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Intervention ogee ow os |£8a8/ Es “aes ose 
No evidence 
: Unofficially Ineligible for (Unofficially 
Convergence Academies Yes WWC Review Yes Yes Yes Yes Ineligible for 
WWC) 
No evidence 
: Unofficially Ineligible for No (Unofficially 
CME ET mes WWC Review Ne Mes i Evidence Ineligible for 
WWC) 
Data-Driven Decision i 
Making and Information unomey Meets Ne No 
fot 4 Yes Standards with Yes Yes Yes Yes : 
Technology Curricula in é (negative) 
Schosle Reservations 
No evidence 
E8TL Teacher Performance Unofficially Ineligible for (Unofficially 
Evaluation System wes WWC Review ue ie tes us Ineligible for 
WWC) 
Unofficially Meets WWC No 
EngageMe P.L.E.A.S.E. Yes Standards without Yes Yes Yes Yes (null) 
Reservations 
ae No evidence 
Every Child Ready (ECR) Yes ase usr Yes Yes No Yes (Unofficially 
DNM WWC) 
F Unofficially Meets WWC 
San eal Yes Standards with Yes Yes Yes Yes at 
Reservations 
Exceptional Coaching for Unofficially Meets WWC No 
Early Language and Literacy Yes® Standards with Yes Yes Yes Yes (null) 
- Enhanced (ExCELL-E) Reservations 
; : Unofficially Meets WWC 
ut tetera NO ane Yes Standards with Yes No Yes Yes Yes 
g Reservations 
sceleuitg Mou Unofficially Meets WWC 
Improvements in Graduation ith No 
and Higher Education for Yes Standards wit Yes Yes Yes Yes (null) 
Tomorrow (FLIGHT) Reservalous 
: Unofficially Meets WWC 
iumnreret eae Yes Standards without Yes Yes Yes No aan 
Reservations 


Abt Associates 


i3: Summary of 67 Evaluations | pg. C-4 


reported results of confirmatory 


Evaluators prespecified and 
impact analyses 


trength of the Impact 


valuation 


Quality implementation 


evaluation 


adequately representative of 


Impact evaluation sample 
population served 


Impact evaluation is considered 


independent 
Evidence that the intervention 


was implemented with 
adequate fidelity 


is 
= 
G 
S 
a= 
SS 
D> 
(72) 
> 
= 
oO 
‘es 
3B 
Ss 
~~ 
wn 
- 
2 
= 
wn 
[2] 
a 


impact on student outcomes 


co) 
= 
() 
Pan 
7) 
c 
& 
~ 
c 
— 
) 
ct) 
° 
iS 
o 
z 
S 
mw 


Intervention Go 
Unofficially Meets WWC No 
InnovateNYC Ecosystem Yes Standards with Yes Yes Yes No i 
Reservations (null) 
Innovations in Earl enetiiaaly Meee wae No 
Mathematics y Yes Standards with Yes No Yes Yes (null 
Reservations 
: ; No evidence 
Integrating English an ae a5 
Language Development Yes re 2 oe fet Yes Yes Yes Yes abe aed 
(ELD) and Science a eee 
WWC) 
Internet-based Reading ss No evidence 
Apprenticeship Improving Yes ee nari Yes Yes Yes Yes (Unofficially 
Science Education (iRAISE) DNM WWC) 
No evidence 
Making Time for What Unofficially Ineligible for (Unofficially 
Matters Most es WWC Review Mesoyi cs es Ne Ineligible for 
WWC) 
New England Network for Unofficially Meets WWC No 
Personalization and Yes Standards with Yes Yes Yes Yes (nul) 
Performance (NETWORK) Reservations 
No evidence 
Unofficially Ineligible for (Unofficially 
Oakland Accelerates Yes WWC Review Yes Yes Yes Yes Ineligible for 
WWC) 
Ounce of Prevention Fund Unofficially Meets WWC No 
Professional Development Yes Standards with Yes Yes Yes Yes (null) 
Initiative (PDI) Reservations 
ata Unofficially Meets WWC 
fg eye SEM Tnnatve Yes Standards with Yes Yes Yes No ath 
Reservations 
No evidence 
Problem-Based Learning is ; No No (No 
(PBL) My Nowplesperiied alialises wes Evidence | Evidence Ld prespecified 
analyses) 
Unofficially Meets WWC No 
PTA Comunitario Yes Standards with Yes No Yes Yes (null) 
Reservations 
: ie No evidence 
Public School Choice Unofficially Does Not Meet _ 
ie Yes Yes No Yes Yes (Unofficially 
Initiative (PSCI) WWC Standards DNM WWC) 
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Responding Effectively to ; 
; ea No evidence 
Assessments with vee Unofficially Does Not Meet Ves No Ves ves (Unofiicially 
Curriculum and Teaching WWC Standards 
(REACT) DNM WWC) 
a eri esarene ral Unofficially Meets WWC a 
ga 9 Yes? Standards without Yes Yes Yes Yes 
Excellence: New Teacher Bieccivalions (null) 
Training (NTT) 
Rural Math Excellence im) HS ie No No 
‘ Yes Standards with No Yes Yes : 
Partnership (RMEP) Racervations Evidence (null) 
Unofficially Meets WWC No 
School of One Yes Standards without Yes Yes Yesh Yes (null) 
Reservations 
Schools to Watch (STW): Unofficially Meets WWC No 
School Transformation Yes Standards with Yes Yes Yes Yes (nul) 
Network Project Reservations 
Science, Technology, ci 
Engineering and Math aay eG No 
; Yes Standards with Yes Yes Yes Yes 
Education for the 21st Bacan/aione (null) 
Century (STEM21) 
Spheres of Proud Unofficially Meets WWC 
Achievement in Reading for Yes Standards without Yes Yes Yes Yes Yes 
Kids (SPARK) Program Reservations 
STEM Learning Unofficially Meets WWC No 
Opportunities Providing Yes Standards without Yes Yes Yes No (null) 
Equity (SLOPE) Reservations 
, Unofficially Meets WWC No 
va es ea Yes Standards with Yes Yes Yes No (null) 
Reservations 
tog a No evidence 
Mee oe TEC ys ee — yes | Yes Yes Yes (Unofficially 
sic aad, DNM WWC) 
No evidence 
Transforming Teacher Talent Unofficially Ineligible for y N (Unofficially 
(13) System te WWC Review Hees) | ics = i Ineligible for 
WWC) 
Unofficially Meets WWC 
eda cin poet Yes Standards with Yes Yes Yes No Yes 
g Reservations 
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No evidence 
; i Unofficially Ineligible for (Unofficially 
We Are A Vill Yes’ : Yes No Yes Yes a 
eine i WWC Review Ineligible for 
WWC) 
Unofficially Meets WWC No 
Write Up! Yes! Standards with Yes Yes Yes No 
> (null) 
Reservations 


Total number of 
evaluations meeting 


criteria 
(N=67 evaluations) 


Notes: 

*KIPP reported 16 of 18 prespecified analyses. 

> Evaluation findings included here are from the RCT design study only and do not include findings from an RD study that was 
also conducted by the evaluator. 

° AVID reported 3 of 4 prespecified analyses. 

TAAL reported 1 of 2 prespecified analyses. 

° ExCELL-E reported 4 of 5 prespecified analyses. 

‘Our review of the findings reported by the evaluation of PBL concludes that they are considered independent, are Unofficially 
Ineligible for WWC Review, and included a sample that is adequately representative of those served by i3. 

2 Rio Grande Valley Center for Teaching and Leading Excellence: New Teacher Training (NTT) reported 5 of 6 prespecified 
analyses. 

" School of One reported 2 analyses; 1 of these 2 was considered independent. 

'We Are A Village reported 1 of 3 prespecified analyses. 

‘Write Up! reported 3 of 4 prespecified analyses. 
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Appendix D: Findings Reported by 67 i3 Impact Evaluations from 


Prespecified Analyses of Student Academic Outcomes 


This appendix presents the findings from analyses that were found to Unofficially Meet WWC Standards with or 
without Reservations. Each row refers to an impact of an intervention versus the comparison condition for a 
particular outcome domain and educational level. The comparison condition is often denoted as Business as Usual 
unless more details were needed to distinguish between different impacts from the same i3 evaluation. Business as 
Usual reflects whatever educational inputs were in place in the control or counterfactual environment and 
therefore varies across locations and evaluations. The effect size column reports the magnitude of the impact in 
standard deviation units using Hedges’ g to calculate the standardized effect sizes, as described in the WWC 
Procedures and Standards Handbook, version 3.0, Appendix F (pp. F.1—-G.12). Findings that are statistically 
significant at the 5 percent level are indicated with an asterisk (*). 


Outcome Domain Unofficial 

Intervention Comparison (see Appendix B) Educational Level WWC Evidence Rating Effect Size 
Scale-up 
KIPP 

ae wos English Language Unofficially Does Not Meet 
KIPP Admission No KIPP Admission Ati Elementary School WWC Standards - 

ba a English Language Unofficially Does Not Meet 
KIPP Admission No KIPP Admission sii Elementary School WWC Standards - 
New KIPP Schools | No KIPP Enger eaatede Middle School noietaly Mees WE ||. aoa! 

Arts Standards with Reservations 
ee ee Unofficially Meets WWC 
KIPP Admission No KIPP Admission 9 dine a Middle School Standards without +0.143* 
Reservations 
KIPP Charter English Language Middle and High Unofficially Meets WWC rm 
Schools Bisness aoe Arts School Standards with Reservations aes 
No KIPP and No 
eal Non-KIPP Charter English Language Middle and High 
Led by a KIPP- Not Reported - 
a Led by a KIPP- Arts School 
Trained Principal me 
Trained Principal 

KIPP Charter English Language Unofficially Meets WWC 
Schools Eisnessas tsi Arts Eig seie Standards with Reservations aes 
Option of Continuing | No Option of English Language Unofficially Meets WWC ‘ 
KIPP Continuing KIPP Arts pligse ee! Standards with Reservations a 
Option of Continuing | No Option of English Language Hich School Unofficially Does Not Meet _ 
KIPP Continuing KIPP Arts 9 WWC Standards 

ee ae Unofficially Does Not Meet 
KIPP Admission No KIPP Admission Mathematics Elementary School WWC Standards - 

me ae Unofficially Does Not Meet 
KIPP Admission No KIPP Admission Mathematics Elementary School WWC Standards - 
New KIPP Schools | No KIPP Mathematics Middle School Snomeleh) Meet WWE |). soe 

Standards with Reservations 
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Outcome Domain 


Unofficial 


Intervention Comparison (see Appendix B) Educational Level WWC Evidence Rating Effect Size 
Unofficially Meets WWC 
KIPP Admission No KIPP Admission Mathematics Middle School Standards without +0.176* 
Reservations 
KIPP Charter . : Middle and High Unofficially Meets WWC 
Schools Eishesealstel Mathemales School , Standards a Reservations bicidaad 
No KIPP and No 
Non-KIPP Charter | on-KIPP Charter Middle and High 
Led by a KIPP- Mathematics Not Reported - 
Trained Principal Bee ihe AEP oe 
Trained Principal 
oe et Business as Usual Mathematics High School enone) meee sbhie +0.273* 
Schools Standards with Reservations 
Option of Continuin No Option of : : Unofficially Meets WWC 
ee : sai KIPP Mathemalis Highoeniont Standards i Reservations mae 
Option of Continuing | No Option of ' ; Unofficially Does Not Meet 
ine : coninlna KIPP Maemalies Hensehoe WWC Standards - 
Reading Recovery 
. . English Language Unofficially Meets WWC 
Reading Recovery? | Business as Usual ae Elementary School Standards without +0.503* 
Reservations 
Success for All (SFA) 
English Language Unofficially Meets WWC 
SFA Business as Usual die Elementary School Standards without +0.159 
Reservations 
. English Language Unofficially Meets WWC 
SFA Business as Usual Ai Elementary School Standards without +0.075 
Reservations 
English Language Unofficially Meets WWC 
SFA Business as Usual dig Elementary School Standards without +0.075 
Reservations 
. English Language Unofficially Meets WWC 
SFA Business as Usual on Elementary School Standards without +0.033 
Reservations 
Teach for America (TFA) 
Unofficially Meets WWC 
TFA Business as Usual | English Language | Early Childhood and Standards without +0.028 
Arts Elementary School Reservations 
Unofficially Meets WWC 
TFA Business as Usual Early Childhood and Standards without +0.05168 
Mathematics Elementary School Reservations 


°° The update to the report would not have changed our overall assessment of the findings. 
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Outcome Domain Unofficial 
Intervention Comparison (see Appendix B) | Educational Level WWC Evidence Rating Effect Size 


Validation 
Advanced ASSET Professional Development 
No Current 
Advanced ASSET | advanced ASSET 
pices aie Professional 
Devlopmen aa Development after Science Elementary School poe lated bide +0.334 
Prior Experience j ; Standards with Reservations 
: : ; Prior Experience 
with Science: It's . f ins 
Flcnicnta with Science: It's 
y Elementary 
Advanced ASSET 
Professional No Current or Prior 
Development after State-wide Science ; Unofficially Meets WWC 
Experience with Curriculum or Splente Eemnettan Seige! Standards with Reservations ae 
Science: It's Training Initiatives 
Elementary 
The Baby Family and Child Education (FACE) Program 
General Unofficially Does Not Meet 
Baby FACE Program | Business as Usual dehicvemes Early Childhood WWC Standards - 
General Unofficially Does Not Meet 
Baby FACE Program | Business as Usual eeavcnici Early Childhood WWC Standards _ 
Children’s Literacy Initiative (CLI) Program 
Paglichanaage Unofficially Meets WWC 
CLI Business as Usual 9 ‘es ae Elementary School Standards without +0.207* 
Reservations 
Sadicnianauade Unofficially Meets WWC 
CLI Business as Usual g ree guag Elementary School Standards without +0.138 
Reservations 
=aalenianaiaae Unofficially Meets WWC 
CLI Business as Usual 9 ie our Elementary School Standards without +0.047 
Reservations 
Sadien anauage Unofficially Meets WWC 
CLI Business as Usual 9 ie oueg Elementary School Standards without +0.008 
Reservations 
Salen landuage Unofficially Meets WWC 
CLI Business as Usual 9 we oe Elementary School Standards without -0.014 
Reservations 
Collaborative Strategic Reading (CSR) 
each ianauade Unofficially Meets WWC 
CSR Business as Usual 9 re a Middle School Standards without +0.073 
Reservations 
=dalcnianiaae Unofficially Meets WWC 
CSR Business as Usual | “9 pe mere Middle School Standards without +0.039 
Reservations 
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Outcome Domain 


Unofficial 


Intervention Comparison (see Appendix B) Educational Level WWC Evidence Rating Effect Size 
Banieh eanauade Unofficially Meets WWC 
CSR Business as Usual g iis guag Middle School Standards without +0.013 
Reservations 
CSR - School Wide | Business as Usual EngiisnTanauade Middle School uneme ny ect bis +0.025 
Arts Standards with Reservations 
CSR - School Wide | Business as Usual engiisy Language Middle School Hnernicialy ve wi? -0.056 
Arts Standards with Reservations 
CSR - School Wide | Business as Usual Engisvlanguege Middle School Hnorieaty ee bli -0.090 
Arts Standards with Reservations 
College Readiness Program (CRP) 
; General Unofficially Meets WWC 
b * 
cid evn eee Achievement teenies) Standards with Reservations ay 
: General : Unofficially Meets WWC 
b * 
oR? BuSesie yaa Achievement Hg >enoe! Standards with Reservations ae 
College-Ready Writers Program (CRWP) 
. : . Unofficially Meets WWC 
CRWP Business as Usual Engisnaiatade Medes Standards without +0.201* 
Arts School : 
Reservations 
Diplomas Now 
Educational : : Unofficially Meets WWC 
Diplomas Now Business as Usual progress and sd oe Standards without +0.054 
attainment Reservations 
Educational ; ; Unofficially Meets WWC 
Diplomas Now Business as Usual progress and Made snag Standards without +0.026 
; School : 
attainment Reservations 
Educational ‘ : Unofficially Meets WWC 
Diplomas Now Business as Usual progress and Miene- ang Hig Standards without +0.003 
; School ; 
attainment Reservations 
enhancing Missouri's Instructional Networked Teaching Strategies (eMINTS) 
Paulehianaiaas Unofficially Meets WWC 
eMINTS Business as Usual g Ais guag Middle School Standards without -0.035 
Reservations 
Enulish banguade Unofficially Meets WWC 
eMINTS Business as Usual 9 i eee Middle School Standards without 0.042 
Reservations 
; ; Unofficially Meets WWC 
ean Business as Usual English Language Middle School Standards without -0.075 
Teach Arts : 
Reservations 
Unofficially Meets WWC 
eMINTS Business as Usual Mathematics Middle School Standards without +0.130 
Reservations 
Unofficially Meets WWC 
eMINTS Business as Usual Mathematics Middle School Standards without +0.048 
Reservations 
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Outcome Domain 


Unofficial 


Intervention Comparison (see Appendix B) | Educational Level WWC Evidence Rating Effect Size 
Unofficially Meets WWC 
al eine Business as Usual Mathematics Middle School Standards without +0.168* 
Reservations 
GO College (An Enhanced Version of Talent Search) 
GO College Whole , Unofficially Meets WWC 
School Program with BO Colepe Wihele:. |’ nglsitanguege High School Standards without +0.027 
School Program Arts 
Learning Community Reservations 
GO College Whole Unofficially Meets WWC 
School Program with GO Cateye Wine Mathematics High School Standards without +0.040 
; | School Program 
Learning Community Reservations 
GO College Whole Academic Unofficially Meets WWC 
School Program with pois Wee performance in High School Standards without +0.045 
; School Program 
Learning Community school Reservations 
pai ae GO College Whole ewe , Unofficially Meets WWC 
School Program with progress and High School é -0.006 
School Program Standards with Reservations 
Learning Community attainment 
Leadership Assistance for Science Education Reform (LASER) 
Unofficially Meets WWC 
LASER Business as Usual Science Elementary School Standards without -0.002 
Reservations 
Unofficially Meets WWC 
LASER Business as Usual Science Middle School Standards without -0.049 
Reservations 
Northeast Tennessee College and Career Ready Consortium 
Educational ~ 
Consortium¢ Business as Usual progress and High School oe hg ne +0.084* 
Standards with Reservations 
attainment 
Educational a 
Consortium* Business as Usual progress and High School dnemelsy ee Mle +0.079* 
Standards with Reservations 
attainment 
: : General . Unofficially Meets WWC 
Cc * 
Consortium Business as Usual icine High School Sivndards wih Reseniations +0.568 
. . General : Unofficially Meets WWC 
Cc *k 
Consortium Business as Usual Aeecenieal High School Seandarde- wih Reservations +0.049 
Reading Apprenticeship (RA) 
Readin English Language eee 
ae Business as Usual g guag High School Standards without +0.147 
Apprenticeship Arts : 
Reservations 
Unofficially Meets WWC 
Reaaing ; Business as Usual Engiso banged High School Standards without +0.139 
Apprenticeship Arts ; 
Reservations 
Reading Enhances Achievement During the Summer (READS) 
Salen lanaisae Unofficially Meets WWC 
READS Business as Usual 9 die ouee Elementary School Standards without +0.008 
Reservations 
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Intervention 


Comparison 


Outcome Domain 
(see Appendix B) 


Educational Level 


Unofficial 
WWC Evidence Rating 


Effect Size 


Saleh anuage Unofficially Meets WWC 
READS Business as Usual 9 ee oe Elementary School Standards without -0.004 
Reservations 
READS - Parental eanlichtanauane Unofficially Meets WWC 
Involvement Business as Usual g ane a Elementary School Standards without -0.044 
(READS - Pl) Reservations 
Sas : diclig eaciutanauaas Unofficially Meets WWC 
Sutra ale Business as Usual 9 oe Elementary School Standards without -0.046 
Followup (READS - Arts 
Reservations 
PIF) 
StartSmart K-3 Plus Program 
Unofficially Meets WWC 
Sista Business as Usual EnglisEangiade Elementary School Standards without +0.150* 
Year Arts : 
Reservations 
Unofficially Meets WWC 
evened 00 Business as Usual Engiisntangtage Elementary School Standards without +0,.113* 
Year Arts : 
Reservations 
Unofficially Meets WWC 
Premed) Business as Usual Engisntangyage Elementary School Standards without +0.030 
Year Arts : 
Reservations 
Unofficially Meets WWC 
Ete eed 0) Business as Usual Engistbangiate Elementary School Standards without +0.003 
Year Arts : 
Reservations 
Eyended School Unofficially Meets WWC 
Business as Usual Mathematics Elementary School Standards without +0.092* 
Reservations 
Virginia Initiative for Science Teaching and Achievement (VISTA) 
ioe Teas : Unofficially Meets WWC 
Professional Business as Usual Science Elementary School +0.068 
Standards with Reservations 
Development 
VISTA Teacher sie 
Professional Business as Usual Science Middle School nomen Mees sided -0.103 
Standards with Reservations 
Development 
VISTA Teacher = 
; ; fficially Meets WW 
Professional Business as Usual Science Middle School pneliele es c -0.169 
Standards with Reservations 
Development 
wou Tea ; Unofficially Does Not Meet 
Professional Business as Usual Science High School = 
WWC Standards 
Development 
ae Teaciet ; Unofficially Does Not Meet 
Professional Business as Usual Science High School = 
WWC Standards 
Development 
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Intervention 


| Comparison 


Outcome Domain 
(see Appendix B) 


Educational Level 


Unofficial 
WWC Evidence Rating 


Effect Size 


Development 


The Achievement Network (ANet) 


; English Language Elementary and Unofficially Meets WWC 
d | 
ane Edsinessasiteld Arts Middle School Standards with Reservations meal 
; Elementary and Unofficially Meets WWC 
d 7 
li Blshes=a5-)=ual anenele Middle School Standards with Reservations es 
Advanced Placement (AP) Insight 
. : ‘ Unofficially Meets WWC 
AP Insight Business as Usual Science High School rade dis-with: Receniations +0.054 
; ; . ‘ Unofficially Meets WWC 
AP Insight Business as Usual Science High School Sivadarie- wile Reservations +0.047 
Unofficially Meets WWC 
AP Insight Business as Usual Science High School ey ridardeuilh Becenialions +0.009 
Unofficially Meets WWC 
AP Insight Business as Usual Science High School Seandard with RESenialiOns -0.004 
Advancement via Individual Determination (AVID) 
— Business as Usual | English Language Middle and High Unofficially Does Not Meet 7 
; Arts School WWC Standards 
intervention 
aa Business as Usual Mathematics ae Hpi ety odes Nees - 
; School WWC Standards 
intervention 
AVID-core-plus Educational Middle and High 
schoolwide Business as Usual progress and School Not Reported - 
intervention attainment 
aati Business as Usual ara be High School Hpbi aty Dees NetMest - 
pee 9 WWC Standards 
intervention attainment 
Around the Corner (ATC) 
2 ; : Unofficially Meets WWC 
Afoundine Cane: Existing SFA English Language | Early Childhood and Siandanis anhalt 40.099 
Program Arts Elementary School 
Reservations 
Arts Achieve: Impacting Student Success in the Arts 
. English Language Elementary and Unofficially Meets WWC 
e +0, 
BHSOGIIEVE Businesses Leta Arts Middle School Standards with Reservations me 
Arts Achievee Business as Usual EnglistiPangiage High School some ny — “a +0.013 
Arts Standards with Reservations 
Arts for Learning (A4L) Lessons 
Esuienanausas Unofficially Meets WWC 
A4L Business as Usual 9 dis an Elementary School Standards without +0.047 
Reservations 
A4L Business as Usual gis tanauege Elementary School Not Reported - 


Arts 
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Intervention 


Comparison 


Outcome Domain 
(see Appendix B) 


Educational Level 


Unofficial 
WWC Evidence Rating 


Effect Size 


Bay State Reading Institute (BSRI) 


Bay State Reading English Language Unofficially Meets WWC 
Institute Bislies ae aa Arts Bema en00) Standards with Reservations a 
Boston Teacher Residency (BTR) 
Newly Hired BTR , English Language Elementary and Unofficially Meets WWC 
Graduates eee Arts Middle School Standards with Reservations mi 
Newly Hired BTR : : Elementary and Unofficially Meets WWC 
Graduates BiSieeee ial Mathemales Middle School Standards with Reservations is 
Building Assets, Reducing Risks (BARR) Model 
Educational Unofficially Meets WWC 
BARR Business as Usual progress and High School Standards without +0.227* 
attainment Reservations 


Collaborative Organizational Model to Promote Aligned Support Structures (COMPASS) 


English Language 


Elementary and 


Unofficially Does Not Meet 


vO Behe tele Arts Middle Schoo! WWC Standards - 
CollegeYES 
. . : Unofficially Does Not Meet 
CollegeYES Business as Usual Science High School WWC Standards - 
Convergence Academies (CA) 
, English Language Elementary and Unofficially Ineligible for _ 
os Business as ste! Arts Middle School WWC Review 
: English Language : Unofficially Ineligible for 
CA Business as Usual shri High School WWC Review - 
English Language Unofficially Ineligible for _ 
CA Business as Usual ois High School WWC Review 
Elementary and Unofficially Ineligible for 
CA Business as Usual Mathematics Middle School WWC Review - 
CA Business as Usual Mathematics High School Bnew) Delneteh - 
: WWC Review 
: Unofficially Ineligible for 
CA Business as Usual Mathematics High School WWC Review - 
Educational es se 
: Unofficially Ineligible for 
CA Business as Usual progress and High School WWC Review - 
attainment 
Educational Ss = 
Unofficially Ineligible for 
CA Business as Usual progress and High School WWC Review - 
attainment 
Eeatona Unofficially Ineligible for 
CA Business as Usual progress and High School WWC Review - 
attainment 
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Intervention 


Comparison 


Outcome Domain 
(see Appendix B) 


Educational Level 


Unofficial 
WWC Evidence Rating 


Effect Size 


Curriculum 2.0 


Curriculum 2.0 


Business as Usual 


English Language 


Elementary School 


Unofficially Ineligible for 


Arts WWC Review 
; . English Language Unofficially Ineligible for 
Curriculum 2.0 Business as Usual aie Elementary School WWC Review 
; ; English Language Unofficially Ineligible for 
Curriculum 2.0 Business as Usual er Elementary School WWC Review 
: . English Language Unofficially Ineligible for 
Curriculum 2.0 Business as Usual ane Elementary School WWC Review 
Unofficially Ineligible for 
Curriculum 2.0 Business as Usual Mathematics Elementary School - 
WWC Review 


Data-Driven Decision 


Data Driven 


Making and Information Technology Curricula in Schools 


Business as Usual 


English Language 


Elementary School 


Unofficially Does Not Meet 


Decision Making Arts WWC Standards 
sas Bivel ; Business as Usual Mathematics Middle School Unemcialy ee wp -0.096* 
Decision Making Standards with Reservations 
E3TL Teacher Performance Evaluation System 
olay Buti a ticuel English Language Elementary and Unofficially Ineligible for _ 
vials pene Arts Middle School WWC Review 
Evaluation System 
era Bicneeeae gia English Language Elementary and Unofficially Ineligible for _ 
esses Arts Middle School WWC Review 
Evaluation System 
Bali Teaenet ; Elementary and Unofficially Ineligible for 
Performance Business as Usual Mathematics ; ; - 
Middle School WWC Review 
Evaluation System 
ais leather Busi isi ile thcnial Elementary and Unofficially Ineligible for _ 
erformance usiness as Usual athematics Middle School WWC Review 
Evaluation System 
EngageMe P.L.E.A.S.E. 
EngageMe English Language Hnariceny Meets We 
92g Business as Usual 9 i Middle School Standards without -0.053 
P.L.E.A.S.E. Arts 
Reservations 
c M eauleh Warauaae Unofficially Meets WWC 
Meer Business as Usual g guag High School Standards without +0.125 
P.L.E.A.S.E. Arts f 
Reservations 
Enaanelle Unofficially Meets WWC 
gag Business as Usual Mathematics Middle School Standards without -0.010 
P.L.E.A.S.E. é 
Reservations 
Endanalle Unofficially Meets WWC 
ae Business as Usual Mathematics High School Standards without +0.239 
P.L.E.A.S.E. 
Reservations 
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Intervention 


Comparison 


Outcome Domain 
(see Appendix B) 


Educational Level 


Unofficial 
WWC Evidence Rating 


Effect Size 


Every Child Ready (ECR) 


Every Child Ready! 


Business as Usual 


English Language 


Early Childhood 


Unofficially Does Not Meet 


Arts WWC Standards 
Every Child Ready’ | Business as Usual cit oma Early Childhood De ai es - 
Every Child Ready! | Business as Usual eee Early Childhood ae = beta - 
Every Child Ready! | Business as Usual cal meal Early Childhood lal a cis - 
Every Child Ready! | Business as Usual Mathematics Early Childhood abi - pA - 


Everyday Arts for Special Education (EASE) 


English Language 


Unofficially Meets WWC 


EASE Business as Usual Aris Elementary School i civdaidis swith Recenmaiions +0.419 
EASE Business as Usual Mathematics Elementary School Unofnicialy Meets wwe +0.005 
Standards with Reservations 
Exceptional Coaching for Early Language and Literacy - Enhanced (ExCELL-E) 
ExCELL-E Business as Usual engisi language Early Childhood sneiey Meets bide -0.008 
Arts Standards with Reservations 
English Language | Early Childhood and Unofficially Meets WWC 
Seale Bieiees ee tal Arts Elementary School Standards with Reservations suis? 
English Language | Early Childhood and Unofficially Meets WWC 
pious sieslcaal acta Arts Elementary School Standards with Reservations is 
; English Language | Early Childhood and Unofficially Does Not Meet 
pica Busness 5 Valet Arts Elementary School WWC Standards 
ExCELL-E Business as Usual engisniLanguege® | Eary-euihooeane Not Reported - 
Arts Elementary School 
The Expository Reading and Writing Course (ERWC) 
English Language Unofficially Meets WWC ; 
ERWC Business as Usual Ate High School Geandarde with REcenialions +0,133 
Facilitating Long-Term Improvements in Graduation and Higher Education for Tomorrow (FLIGHT) 
Academic i 
FLIGHT Business as Usual performance in Mole ene Eig) ehetislaly Hee me -0.056 
school Standards with Reservations 
school 
Florida Master Teacher Initiative (FMTI) 
Eanrenenauade Unofficially Meets WWC 
FMTIs Business as Usual 9 hai guag Elementary School Standards without -0.038 
Reservations 
FMTI with Early ae 
Childhood Master's | Business as Usual Engst anguade Elementary School ponealy Meets wwe +0.170 
Arts Standards with Reservations 
Programs 
Unofficially Meets WWC 
FMTIg Business as Usual Mathematics Elementary School Standards without -0.038 
Reservations 
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Outcome Domain Unofficial 


Intervention Comparison (see Appendix B) | Educational Level WWC Evidence Rating Effect Size 
FMTI with Early ae 
; lly Meets WW 
Childhood Master's | Business as Usual Mathematics Elementary School oe y i e -0.230 
Standards with Reservations 
Programs 
InnovateNYC Ecosystem 
Math Apps" Business as Usual Mathematics Middle School snomoly ese We +0.075 
Standards with Reservations 
Innovations in Early Mathematics 
Innovations in Early ; Early Childhood and Unofficially Meets WWC 
Mathematics Busse as Usual Mathematics Elementary School Standards with Reservations lie 
Innovations in Early Early Childhood and Unofficially Meets WWC 
Mathematics pee vaemane: Elementary School Standards with Reservations aes 
Integrating English Language Development (ELD) and Science 
Exploratorium - fee 
Professional Business as Usual English Language Elementary School spony ineugiile cl - 
Arts WWC Review 
Development 
Exploratorium og a 
Professional Business as Usual Science Elementary School Pnomcaly pele o - 
WWC Review 
Development 
Internet-based Reading Apprenticeship Improving Science Education (iRAISE) 
; English Language Unofficially Does Not Meet 
iIRAISE Business as Usual Ais High School WWC Standards 
Making Time for What Matters Most 
Making Time for English Language Unofficially Ineligible for 
What Matters Most Busnes eet Arts High sctioo WWC Review 
Making Time for English Language Unofficially Ineligible for 
What Matters Most Bipiene os ata Arts High sete WWC Review 7 
Making Time for : . : Unofficially Ineligible for 
What Matters Most Business as Usual Mathematics High School WWC Review 
Making Time for F . . Unofficially Ineligible for 
What Matters Most! Business as Usual Science High School WWC Review - 
Making Time for Educational Uinoteesdiiy Inalcibletar 
What Matters Most'_ | Business as Usual progress and High School y 9 - 
F WWC Review 
attainment 
Making Time for Educational Unoriealy Inelinible for 
What Matters Most! | Business as Usual progress and High School y 9 - 
F WWC Review 
attainment 
Making Time for Bucwasens lisidl cuaatee High School Unofficially Ineligible for _ 
What Matters Most progr 9 WWC Review 
attainment 
New England Network for Personalization and Performance (NETWORK) 
NETWORK Business as Usual Engen Language High School ssid: eel we -0.120 
Arts Standards with Reservations 
NETWORK Business as Usual Mathematics High School momen Heer ie +0.062 
Standards with Reservations 
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Outcome Domain Unofficial 
Intervention Comparison (see Appendix B) | Educational Level WWC Evidence Rating Effect Size 
Educational ae 
NETWORK Business as Usual progress and High School snemnelaty Meets we +0.041 
. Standards with Reservations 
attainment 
Educational - 
NETWORK Business as Usual progress and High School snore Meets wie -0.055 
. Standards with Reservations 
attainment 
Educational as 
NETWORK Business as Usual progress and High School ore Meets we -0.252 
Standards with Reservations 
attainment 
Oakland Accelerates 
: English Language Unofficially Ineligible for 
Oakland Accelerates | Business as Usual roe High School WWC Review - 
Oakland Accelerates | Business as Usual Mathematics High School Hnotinely nekgile io - 
WWC Review 
; General Unofficially Does Not Meet 
Oakland Accelerates | Business as Usual Aen High School WWC Standards - 
Educational oe 
Unofficially Does Not Meet 
Oakland Accelerates | Business as Usual progress and High School WWC Standards - 
attainment 
eateaponal Unofficially Ineligible for 
Oakland Accelerates | Business as Usual progress and High School y g - 
F WWC Review 
attainment 
Eaueatenal Unofficially Ineligible for 
Oakland Accelerates | Business as Usual progress and High School y g - 
F WWC Review 
attainment 
Eteanonel Unofficially Ineligible for 
Oakland Accelerates | Business as Usual progress and High School y g - 
F WWC Review 
attainment 
Ounce of Prevention Fund Professional Development Initiative (PDI) 
. English Language Unofficially Meets WWC 
PDI Business as Usual ais Early Childhood Sianderds aul Pesenvaons +0.253 
PDI English Language Unofficially Meets WWC 
Business as Usual ee Early Childhood i adattic quik: Resenanens +0.064 
PDI ; English Language , Unofficially Does Not Meet 
Business as Usual ee Early Childhood WWC Standards - 
PDI ‘ : . Unofficially Does Not Meet 
Business as Usual Mathematics Early Childhood WWC Standards - 
PDI : General : Unofficially Meets WWC 
Business as Ustal Achievement Ealy Cucnord Standards with Reservations oe 
PDI ; General Unofficially Does Not Meet 
Business as Usual ee Early Childhood WWC Standards - 
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Intervention 


Comparison 


Outcome Domain 
(see Appendix B) 


Educational Level 


Unofficial 
WWC Evidence Rating 


Effect Size 


Pathways to STEM Initiative (PSI) 


Unofficially Meets WWC 


PSI Business as Usual Science Middle School éeandards with Reservations +0.010 
PSIi Business as Usual Science Middle School Unoineally Meets we -0.056 
Standards with Reservations 
Problem-Based Learning (PBL) 
No prespecified analyses‘ - 
PTA Comunitario 
PTA Comunitario Business as Usual EngiiswLanguade Elementary School Bnemncialy ca we +0.025 
Arts Standards with Reservations 
PTA Comunitario Business as Usual English aniguage Middle School Uneticially igi me -0.003 
Arts Standards with Reservations 
a , English Language Unofficially Does Not Meet 
PTA Comunitario Business as Usual Ais High School WWC Standards 
PTA Comunitario Business as Usual Mathematics Elementary School lias lipid bling -0.021 
Standards with Reservations 
PTA Comunitario Business as Usual Mathematics Middle School eee fe ise +0.008 
Standards with Reservations 
PTA Comunitario Business as Usual Mathematics High School pueinially Dees Nat Meet - 


WWC Standards 


Public School Choice Initiative (PSCl) 


PSCI: Focus picneceas isial English Language Elementary, Middle Unofficially Does Not Meet _ 
Schools Arts and High School WWC Standards 
PSCI: Focus Busicseaelicual English Language Elementary, Middle Unofficially Does Not Meet _ 
Schools Arts and High School WWC Standards 
ae : English Language Elementary, Middle Unofficially Does Not Meet 
PSCI: Relief Schools | Business as Usual Ait and High School WWC Standards = 
a . English Language Elementary, Middle Unofficially Does Not Meet 7 
PSCI: Relief Schools | Business as Usual Ai and High School WWC Standards 
PSCI: Focus Bacnessaslstial Kathamatics Elementary, Middle Unofficially Does Not Meet _ 
Schools and High School WWC Standards 
PSCI: Focus saeeesee aia hiatheaatics Elementary, Middle Unofficially Does Not Meet _ 
Schools and High School WWC Standards 
won Elementary, Middle Unofficially Does Not Meet 
PSCI: Relief Schools | Business as Usual Mathematics and High School WWC Standards 
aE ; Elementary, Middle Unofficially Does Not Meet 
PSCI: Relief Schools | Business as Usual Mathematics and High School WWC Standards - 


Responding Effectively to Assessments with Curriculum and Teaching (REACT) 


English Language 


Elementary, Middle 


Unofficially Does Not Meet 


one Bish es use! Arts and High School WWC Standards : 
Elementary, Middle Unofficially Does Not Meet 
REACT Business as Usual Mathematics and High School WWC Standards 7 
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Outcome Domain Unofficial 


Intervention Comparison (see Appendix B) | Educational Level WWC Evidence Rating Effect Size 


Rio Grande Valley Center for Teaching and Leading Excellence: New Teacher Training (NTT) 
= aulichanaiaae Unofficially Meets WWC 
NTT Business as Usual 9 Ais ae Elementary School Standards without -0.039 
Reservations 
Paulichiavaigae Unofficially Meets WWC 
NTT Business as Usual 9 pe a Middle School Standards without 0.128 
Reservations 
Unofficially Meets WWC 
NTT Business as Usual Mathematics Elementary School Standards without +0.083 
Reservations 
Unofficially Meets WWC 
NTT Business as Usual Mathematics Middle School Standards without -0.298 
Reservations 
NTT Business as Usual Science Middle School Not Reported - 
NTT Business as Usual Social Studies Middle School Uneticially te leg -0.120 
Standards with Reservations 
Rural Math Excellence Partnership (RMEP) 
RMEP Project Business as Usual Mathematics ie ane iit snemelaly Hes we -0.950 
School Standards with Reservations 
School of One 
Unofficially Meets WWC 
School of One Business as Usual Mathematics Middle School Standards without +0.035 
Reservations 
Unofficially Meets WWC 
School of One! Business as Usual Mathematics Middle School Standards without -0.022 
Reservations 
Schools to Watch (STW): School Transformation Network Project 
STW Business as Usual English Language Middle School enemie nly Meets Wine -0.059 
Arts Standards with Reservations 
STW Business as Usual Mathematics Middle School mutica Wests Wie 0.145 
Standards with Reservations 
Science, Technology, Engineering and Math Education for the 21st Century (STEM21) 
. ; : icially Meets WW! 
STEM21 Academy Business as Usual Science High School Suotibaly as . +0.012 
Standards with Reservations 
Spheres of Proud Achievement in Reading for Kids (SPARK) Program 
Paulich vanauasé Unofficially Meets WWC 
SPARK Business as Usual 9 Aits oe Elementary School Standards without +0.355* 
Reservations 
easlch varauane Unofficially Meets WWC 
SPARK Business as Usual 9 Ais oe Elementary School Standards without +0.226* 
Reservations 
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Intervention 


Comparison 


Outcome Domain 
(see Appendix B) 


Educational Level 


Unofficial 
WWC Evidence Rating 


Effect Size 


STEM Learning Opportunities Providing Equity (SLOPE) 


SLOPE: Algebra 1 


Unofficially Meets WWC 


Drop-in Units! Business as Usual Mathematics Middle School Geandatds with Racenialiois -0.101 
SLOPE: Algebra 1 
A Slane Unofficially Meets WWC 
9 Business as Usual Mathematics Middle School Standards without -0.147 
Awareness and Bi cenaticne 
Math Proficiency 
(C.A.M.P.)™ 
STEM Summer Learning with VEX Robotics 
STEM Summer No Participation in . Unofcially Meets WWC 
Learning Program Any Summer Mathematics Middle School 0.000 
; : F Standards with Reservations 
with VEX Robotics" | Learning Program 
Texas Tech University “Tech Teach” Program 
; English Language Elementary and Unofficially Does Not Meet 
Teen leaal Business ost Arts Middle School WWC Standards 7 
Competency Based : ; pas 
an Middle and High Unofficially Does Not Meet 
Training and Math Math Forward only Mathematics School WWC Standards - 
Forward 
Elementary, Middle Unofficially Meets WWC - 
TechTeach Business as Usual Mathematics and High School Seandardle wih Reservations 0.134 
Transforming Teacher Talent (t3) System 
English Language Elementary, Middle Unofficially Does Not Meet 
0 = 
PSE SYSIen. || BUSINESS ae sel Arts and High School WWC Standards 
Elementary and Unofficially Does Not Meet 
0 = 
Aspire t3 System Business as Usual Mathematics Middle School WWC Standards 
Turnaround with Increased Learning Time (TILT) 
TILT Business as Usual EngiisyLanguage Middle School Hnemcialy ied wie +0.384* 
Arts Standards with Reservations 
TILTP Business as Usual Mathematics Middle School pone peels line +0.021 
Standards with Reservations 
TILTP Business as Usual Science Middle School poi Mes Wve +0.089 


Standards with Reservations 


We Are A Village 


We Are A Village 


We Are A Village 


Business as Usual 


Business as Usual 


English Language 
Arts 
English Language 
Arts 


Early Childhood and 
Elementary School 
Early Childhood and 
Elementary School 


Unofficially Ineligible for 
WWC Review 


Not Reported 


69 


Summary findings for TechTeach are not reported in Appendix C because the modal evidence rating across the findings 


reported is Unofficially Does Not Meet WWC Standards. One of the three findings is reported here because that finding 
received an evidence rating of Unofficially Meets WWC Standards with Reservations. 
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Outcome Domain Unofficial 


Intervention Comparison (see Appendix B) | Educational Level WWC Evidence Rating Effect Size 
We Are A Village We Are A Village English Language | Early Childhood and Nor RenoHad 2 
(Year 2) (Year 1) Arts Elementary School P 
Write Up! 
; English Language Unofficially Meets WWC 
! f 
Write Up!s Business as Usual aie Elementary School Geindards wih Recenter 0.048 
: English Language Unofficially Meets WWC 
Ig . 
Write Up! Business as Usual er Elementary School Giudice ail Resenvalione 0.198 
. . English Language Middle and High Unofficially Meets WWC 
Iq 
MEE Business eetstel Arts school Standards with Reservations te 
Write Up!s Business as Usual Englisti Language idole ane Hight Not Reported - 
Arts school 
Notes: 


* Evaluation findings included here are from the RCT design study only and do not include findings from an RD study that was 
also conducted by the evaluator. 

» College Readiness Program was not implemented with adequate fidelity. 

“Northeast Tennessee College and Career Ready Consortium was not implemented with adequate fidelity. 

* ANet was not implemented with adequate fidelity. 

° Arts Achieve was not implemented with adequate fidelity. 

‘These Every Child Ready analyses were not considered independent. 

3 FMTI, with and without Early Childhood Master’s Program, was not implemented with adequate fidelity. 

: Math Apps was not implemented with adequate fidelity. 

‘Making Time for What Matters Most was not implemented with adequate fidelity. 

Pathways to STEM Initiative was not implemented with adequate fidelity. 

‘See Appendix F for findings from analyses that were not prespecified. 

' This School of One analysis was not considered independent. 

™ SLOPE: Algebra 1 Drop-in Units, with and without Summer C.A.M.P., was not implemented with adequate fidelity. 
" STEM Summer Learning Program with VEX Robotics was not implemented with adequate fidelity 

° Aspire t3 System was not implemented with adequate fidelity. 

® Turnaround with Increased Learning Time was not implemented with adequate fidelity. 

4 Write Up! was not implemented with adequate fidelity. 
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Appendix E: Findings Reported by 21 i3 Impact Evaluations from 
Prespecified Analyses on Non-Academic Outcomes 


Intervention 
Development 


| Comparison 


Outcome 
Domain? 


Arts Achieve: Impacting Student Success in the Arts 


Educational 
Level 


Unofficial 
WWC Evidence Rating 


Elementary, BG 
Arts Achieve? Business as Usual | Art Achievement | Middle, and High lad, iets pac +0.095 
Standards with Reservations 
School 
Everyday Arts for Special Education (EASE) 
EASE Bu sneseke Usual Social/Emotional Elementary Unofficially Meets Wwe 40.179" 
Learning School Standards with Reservations 
Exceptional Coaching for Early Language and Literacy - Enhanced (EXCELL-E) 
aly Seeks oh || seal Caneneee Unofficially Does Not Meet WWC 
ExCELL-E Business as Usual Instructional and Elementary - 
Standards 
Environment School 
Facilitating Long-Term Improvements in Graduation and Higher Education for Tomorrow (FLIGHT) 
FLIGHT Business as Usual Behavior Middle anal Hligh vaemelaly Meets we +0.007 
School Standards with Reservations 
FLIGHT Business as Usual Motivation Made ert uaemealy eee wwe -0.085 
School Standards with Reservations 
Florida Master Teacher Initiative (FMTI) 
Assessment a 
FMTI¢ Business as Usual Informed Eememaly snemealy Meets We -0.082 
: School Standards without Reservations 
Instruction 
sl Ean pesca ient Elementary Unofficially Meets WWC 
Childhood Master's Business as Usual Informed ; +0.076 
. School Standards with Reservations 
Program Instruction 
Classroom 
al wea Instruction: Elementary Unofficially Meets WWC 
Childhood Master's Business as Usual : -0.475 
Classroom School Standards with Reservations 
Programe os 
Organization 
pel MEY , Session Elementary Unofficially Meets WWC 
Childhood Master's Business as Usual Instruction: f : -0.046 
. School Standards with Reservations 
Programs Emotional Support 
Classroom 
a ally Instruction: Elementary Unofficially Does Not Meet WWC 
Childhood Master's Business as Usual . - 
Instructional School Standards 
Program 
Support 
Collaboration Elementary Unofficially Meets WWC 
c =| 
em Eismessos.Uslld Around Instruction School Standards without Reservations oats 
i mally Collaboration Elementary Unofficially Does Not Meet WWC 
Childhood Master's Business as Usual - 
around Instruction School Standards 


Program 


Abt Associates 


i3: Summary of 67 Evaluations 1] pg. E-1 


Outcome Educational Unofficial 
Intervention Comparison Domain? Level WWC Evidence Rating 
Culturally < 
FMTI¢ Business as Usual Responsive Eemenaly Enemy Meets wae -0.046 
School Standards without Reservations 
Instruction 
ey mea : ee) Elementary Unofficially Does Not Meet WWC 
Childhood Master's Business as Usual Responsive = 
School Standards 
Programs Instruction 
Developmentally - 
Elementary Unofficially Meets WWC 
em Bletiess a USiel pperopiiate School Standards without Reservations ae 
Instruction 
om wea Deveppmently Elementary Unofficially Meets WWC 
Childhood Master's Business as Usual Appropriate é -0.087 
School Standards with Reservations 
Programs Instruction 
; Differentiated Elementary Unofficially Meets WWC : 
ull Bismessies Veet Instruction School Standards without Reservations call 
sual wieally Differentiated Elementary Unofficially Meets WWC 
Childhood Master's Business as Usual ; +0.100 
Instruction School Standards with Reservations 
Programe 
ea ened Elementary Unofficially Meets WWC 
FMTI¢ Business as Usual Teaching +0.055 
School Standards without Reservations 
Knowledge 
il idee! = enlace? Elementary Unofficially Meets WWC ‘ 
Childhood Master's Business as Usual Teaching +0.629 
School Standards with Reservations 
Programe Knowledge 
Effective Principal Elementary Unofficially Meets WWC 
c 
eo EYSnceecenaia P-3 Leadership School Standards without Reservations nite 
ul os Effective Principal Elementary Unofficially Does Not Meet WWC 
Childhood Master's Business as Usual ; - 
P-3 Leadership School Standards 
Program 
Emphasis on Bd 
FMTI¢ Business as Usual Higher-order Piementaly snemeraly sided we +0.004 
hae School Standards without Reservations 
Thinking Skills 
Bl auth Ealy Empuas: a Elementary Unofficially Does Not Meet WWC 
Childhood Master's Business as Usual Higher-order - 
ae School Standards 
Programs Thinking Skills 
; Family Elementary Unofficially Meets WWC 
PM: Eusinepeies Usia Partnerships School Standards without Reservations 70S 
al oe Family Elementary Unofficially Does Not Meet WWC 
Childhood Master's Business as Usual ; - 
Partnerships School Standards 
Programe 
Pleven eect Elementa' Unofficially Meets WWC 
FMTI¢ Business as Usual Variety of y ad ; -0.059 
School Standards without Reservations 
Assessments 
Bal wally ereauen oe Elementary Unofficially Meets WWC 
Childhood Master's Business as Usual Variety of / -0.212 
School Standards with Reservations 
Program Assessments 
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Intervention 


Comparison 


Outcome 
Domain? 


Educational 
Level 


Unofficial 
WWC Evidence Rating 


Governance Elementary Unofficially Meets WWC * 
eum Business Usvel Activities School Standards without Reservations aueet 
FMTI with Early oe 

. fficially Meets WW' 

Childhood Master's Business as Usual aula aan Eemeney siete Sis . +6.537* 
Activities School Standards with Reservations 
Program 

Learner-centered Elementary Unofficially Meets WWC 
au Busines as Vata! Instruction School Standards without Reservations aes 
ul amEatly ; Learner-centered Elementary Unofficially Does Not Meet WWC 
Childhood Master's Business as Usual : - 

Instruction School Standards 
Program 
FMTI¢ Business as Usual | Outreach Activities Eiemenaly vneticaly Meets me +0.811* 
School Standards without Reservations 
FMTI with Early e 
soa Elementary Unofficially Meets WWC sp 
Childhood Master's Business as Usual | Outreach Activities Schaal Sndatds valle seniors +2,941 
Program 
Teacher Elementary Unofficially Meets WWC 
c +0.04 
ea Bibtess a Usiel Leadership School Standards without Reservations ae 
FMTI with Early 7 
; ; El t fficially Meets WW' 
Childhood Master's Business as Usual Teachel puna ouee aaa : +0.460* 
Leadership School Standards with Reservations 
Program 
Teaching Elementary Unofficially Meets WWC 
c +0.047 
cul BUSneSs eee Knowledge School Standards without Reservations al 
FMTI with Early 7 
El t fficially Meets WW' 
Childhood Master's Business as Usual eae Ld euoiiee edie ° +0.564* 
Knowledge School Standards with Reservations 
Program 
Trusting i 
FMTI¢ Business as Usual Relationship Eementaty one sed _ +0.024 
School Standards without Reservations 
between Teachers 
FMTI with Early Trusting fa 
é' Elementary Unofficially Meets WWC 
Childhood Master's Business as Usual Relationship School Seandarde with Reservations +0,184 
Programe between Teachers 
InnovateNYC Ecosystem 
; Opinions about Middle and High Unofficially Meets WWC 
d +0. 
CapiAps Eusmess a> sia Technology School Standards with Reservations meee 
. Perceptions about | Middle and High Unofficially Meets WWC 
d 
ates Bushes es Math School Standards with Reservations cee 
Perceptions about | Middle and High | Unofficially Does Not Meet WWC 
d = 
Math Apps Business as Usual Math Schaal Sedaris 
Perceptions about | Middle and High | Unofficially Does Not Meet WWC 
d ~ 
Math Apps Business as Usual Math Schad Siandarde 
Teacher's Use of os 
; Middle and High Unofficially Meets WWC 
d 
Gap Apps Business as Usual | Technology in the Schoo! Seandarde withiReservations +0.187 


Classroom 
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Intervention 


Comparison 


Outcome 
Domain? 


Educational 
Level 


Unofficial 
WWC Evidence Rating 


Teacher’s Use of 


Middle and High 


Unofficially Meets WWC 


d j j 7 
Gap Apps Business as Usual | Technology in the echosl Gianidarde: with Gesenvations 0.260 
Classroom 
Teacher's Use of wd 
Gap Apps! Business as Usual | Technology in the Middle and High | Unofficially Does Not Meet WWC _ 
sk School Standards 
Classroom 
Making Time for What Matters Most 
Making Time for What Busiesens licual Atendance High School Unofficially Ineligible for WWC _ 


Matters Most® 


Review 


Oakland Accelerates 


Oakland Accelerates 


Oakland Accelerates 


Ounce of Prevention Fund Professional Development Initiative (PDI) 


PDI 


PDI 


PDI 


PDI 


PDI 


PDI 


PDI 


PDI 


PDI 


PDI 
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Business as Usual 


Business as Usual 


Business as Usual 


Business as Usual 


Business as Usual 


Business as Usual 


Business as Usual 


Business as Usual 


Business as Usual 


Business as Usual 


Business as Usual 


Business as Usual 


Participation in 
College Entrance 
Placement and 
Performance 
Exams 
Participation in 
College Entrance 
Placement and 
Performance 
Exams 


Child's Social & 
Emotional 
Development 
Classroom 
Organization 
Diverse Learning 
Opportunities 
Diverse Learning 
Opportunities 
Diverse Learning 
Opportunities 
Emotional 
Supports 
Emotional 
Supports 
Facilitation of 
Learning 
Facilitation of 
Learning 
Oral Language 
and Linguistic 
Environment 


High School 


High School 


Early Childhood 


Early Childhood 
Early Childhood 
Early Childhood 
Early Childhood 
Early Childhood 
Early Childhood 
Early Childhood 


Early Childhood 


Early Childhood 


Unofficially Does Not Meet WWC 
Standards 


Unofficially Does Not Meet WWC 
Standards 


Unofficially Meets WWC 
Standards with Reservations 


Unofficially Meets WWC 
Standards with Reservations 


Not Reported 
Not Reported 


Not Reported 


Unofficially Meets WWC 
Standards with Reservations 


Not Reported 


Unofficially Meets WWC 
Standards with Reservations 


Not Reported 


Not Reported 
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Outcome 


| Educational 


Unofficial 


| WWC Evidence Rating 


Intervention Comparison Domain? Level 
Oral Language 
PDI Business as Usual and Linguistic Early Childhood Not Reported - 
Environment 
Oral Language 
PDI Business as Usual and Linguistic Early Childhood Not Reported - 
Environment 
PDI Business as Usual | Program Structure | Early Childhood Not Reported - 
PDI Business as Usual | Program Structure | Early Childhood Not Reported - 
PDI Business as Usual | Program Structure | Early Childhood Not Reported - 
PDI Business as Usual ghey o Early Childhood Not Reported - 
Interactions 
PDI Business as Usual ually ot Early Childhood Not Reported 4 
Interactions 
PDI Business as Usual ually of Early Childhood Not Reported - 
Interactions 
Rio Grande Valley Center for Teaching and Leading Excellence: New Teacher Training (NTT) 
Elementary, a 
Leadership Skills Training | Business as Usual Teac peadet Middle, and High sneticialy heer ms -0.280 
Attrition Standards without Reservations 
School 
Elementary, a 
Skillful Teacher Institute Business as Usual Teac peace Middle, and High query MESS we -2.393 
Attrition Standards without Reservations 
School 
Elementary, 7 
Leadership Skills Training | Business as Usual Teaget Eeaeel Middle, and High Enel Meets we +0.237 
Job Satisfaction School Standards without Reservations 
Elementary, oe 
Skillful Teacher Institute Business as Usual Tear! peaget Middle, and High oneal) MES wine -0.034 
Job Satisfaction Schaal Standards without Reservations 
Elementary, 7 
Leadership Skills Training | Business as Usual Neathet Pavel Middle, and High Haomelaty eta me +0.071 
Self-efficacy cahac Standards without Reservations 
Elementary, ts 
Skillful Teacher Institute Business as Usual Tea cane Middle, and High Haomealy MESS ne -0.395* 
Self-efficacy Schacl Standards without Reservations 
Rural Math Excellence Partnership (RMEP) 
Same students Attitudes toward Middle and High | Unofficially Does Not Meet WWC 
RMEP Project ‘ - 
prior to exposure Math School Standards 
Same students Attitudes toward Middle and High | Unofficially Does Not Meet WWC 
RMEP Project : - 
prior to exposure Math School Standards 
Same students Attitudes toward Middle and High | Unofficially Does Not Meet WWC 
RMEP Project P - 
prior to exposure Math School Standards 
Same students Interest in STEM | Middle and High | Unofficially Does Not Meet WWC 
RMEP Project : - 
prior to exposure Careers School Standards 
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Outcome Educational Unofficial 
Intervention Comparison Domain? Level WWC Evidence Rating 
Same students Interest in STEM | Middle and High | Unofficially Does Not Meet WWC 
RMEP Project . = 
prior to exposure Careers School Standards 
Same students Interest in STEM | Middle and High | Unofficially Does Not Meet WWC 
RMEP Project : - 
prior to exposure Careers School Standards 
Same students Interest in STEM | Middle and High | Unofficially Does Not Meet WWC 
RMEP Project ; - 
prior to exposure Careers School Standards 
Same students Interestin STEM | Middle and High | Unofficially Does Not Meet WWC 
RMEP Project : - 
prior to exposure Careers School Standards 
Same students Interest in STEM | Middle and High | Unofficially Does Not Meet WWC 
RMEP Project ; = 
prior to exposure Careers School Standards 
Same students Interestin STEM | Middle and High | Unofficially Does Not Meet WWC 
RMEP Project : = 
prior to exposure Careers School Standards 
School of One 
Academic e 
School of One Business as Usual Behaviors and Middle School pam Mee we -0.186 
Standards without Reservations 
Attitudes 
School of One Business as Usual Teatiet Middle School phoma) ees Not Meee - 
Fulfillment Standards 
School of One Business as Usual Teeinoiogie! Middle School Not Reported - 
Knowledge 
Science, Technology, Engineering and Math Education for the 21st Century (STEM21) 
Career and - 
STEM21 Academy Business as Usual College-Going High School ale ee me +0.187 
Standards with Reservations 
Interest in STEM 
STEM 21 Academy . Interest in STEM Unofficially Meets WWC 
Afterschool Program! ishiess aeeta Subjects oe Standards with Reservations aie 
STEM Summer Learning with VEX Robotics 
STEM Summer 
STEM Summer Learning | Learning Program Aspirations to oe 
Program with VEX without VEX Study STEM in Middle School Pasi senile +0.233 
Robotics9 Robotics (with Arts College 
or Sports) 
STEM Summer 
STEM Summer Learning | Learning Program Aspirations to 7 
Program with VEX without VEX Study STEM in | Middle School Pees eae +0.068 
Robotics9 Robotics (with Arts College 
or Sports) 
STEM Summer Learning | No Participation in i 
Program with VEX Any Summer Attendance Middle School waomeely ue ae +0.094 
; ; Standards with Reservations 
Roboticss Learning Program 
STEM Summer 
STEM Summer Learning | Learning Program 2 
Program with VEX without VEX — Middle School Pedra ene +0.146 
Roboticss Robotics (with Arts P 
or Sports) 
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Outcome Educational Unofficial 
Intervention Comparison Domain? Level WWC Evidence Rating 
STEM Summer 
STEM Summer Learning | Learning Program ae 
Program with VEX without VEX peice Middle School Pah cea a a 0.173 
Robotics9 Robotics (with Arts P 
or Sports) 
STEM Summer 
STEM Summer Learning | Learning Program = 
Program with VEX without VEX pelea Middle School Peer riaaes sci +0.055 
Robotics9 Robotics (with Arts P 
or Sports) 
STEM Summer 
STEM Summer Learning | Learning Program 
Program with VEX without VEX hina Middle School Paes Saat 0.014 
Robotics9 Robotics (with Arts P 
or Sports) 
Turnaround with Increased Learning Time (TILT) 
TILT* Business as Usual Attendance Middle School Unemncraly Meet me +0.032 
Standards with Reservations 
TILT" Business as Usual etiaet Middle School mnonicially Does Nor Meer Wie - 
Engagement Standards 
We Are A Village 
Early Childhood = fa 
We Are A Village Business as Usual Attendance and Elementary enone neta TE - 
Review 
School 
Early Childhood 
We Are A Village (Year 2) Wenen ViBve Attendance and Elementary Not Reported - 
(Year 1) 
School 
Early Childhood 
We Are A Village (Year 2) Wenn Mave Attendance and Elementary Not Reported - 
(Year 1) 
School 
Early Childhood 7 ae 
We Are A Village’ Business as Usual Behavior and Elementary ey elec ae - 


School 


Review 


Validation 


The Baby Family and Child Education (FACE) Program 


Baby FACE Program 


Business as Usual 


Parent Perception 
of Socio-emotional 
Development 


Early Childhood 


Unofficially Does Not Meet WWC 
Standards 


Baby FACE Program 


Business as Usual 


Parent Perception 
of Socio-emotional 


Early Childhood 


Unofficially Does Not Meet WWC 


Development standalds 
Children’s Literacy Initiative (CLI) Program 
Classroom be 
CLI Business as Usual Literacy Enea shone! iad sida +0.576* 
. School Standards without Reservations 
Environment 
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Outcome Educational Unofficial 
Intervention Comparison Domain? Level WWC Evidence Rating 
Teacher Elementary Unofficially Meets WWC 7 
| Buisiiessias ste! Instruction School Standards without Reservations eli 
enhancing Missouri's Instructional Networked Teaching Strategies (eMINTS) 
eMINTS Business as Usual | 21st Century Skills | Middle School pci) Meee ine +0.025 
Standards without Reservations 
eMINTS Business as Usual | 21st Century Skills | Middle School Bueinaly Meee iNG +0.107 
Standards with Reservations 
eMINTS with Intel Teach | Business as Usual | 21st Century Skills | Middle School wpomgay Mee ee +0.077 
Standards without Reservations 
GO College (An Enhanced Version of Talent Search) 
GO College Whole School | 6 college whole . Unofficially Meets WWC 
Program with Learning Attendance High School / -0.078 
School Program Standards without Reservations 
Community 
Se euse wie Senay GO College Whole : Unofficially Meets WWC 
Program with Learning Behavior High School ; -0.016 
School Program Standards with Reservations 
Community 
GO College Whole School , 
Program with Learning Bo Gallege whale Course Taking High School phaticialy MEeeN Ne +0.031 


Community 
StartSmart K-3 Plus Progr: 


Extended School Year 


School Program 
am 


Business as Usual 


Social Skills 


Elementary 
School 


Standards with Reservations 


SSS Se Se ee 


Unofficially Meets WWC 
Standards without Reservations 


Notes: 


* These outcome domains were established by the evaluators, as these outcomes are outside of the domains defined in 


Appendix B. 


» arts Achieve was not implemented with adequate fidelity. 
° FMTI, with and without Early Childhood Master’s Program, was not implemented with adequate fidelity. 
. Gap Apps and Math Apps were not implemented with adequate fidelity. 

° Making Time for What Matters Most was not implemented with adequate fidelity. 


‘This STEM Education for the 21st Century analysis was not considered independent. 


3 STEM Summer Learning Program with VEX Robotics was not implemented with adequate fidelity. 
_ Turnaround with Increased Learning Time was not implemented with adequate fidelity. 


‘This We Are A Village analysis was not considered independent. 
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Appendix F: Findings Reported by 40 i3 Impact Evaluations from Analyses 


that Were Not Prespecified 


The findings included in this appendix were reported by evaluators but were not from analyses that were 
prespecified. And while they are often from analyses of the same intervention condition, comparison condition, 
outcome domain, and educational level, they differ from those reported in Appendix D and Appendix E along one 
or more dimensions. These dimensions include limiting the analysis to a subgroup of those who were served, to 
those served in a particular year, to those served for a particular amount of time, or estimating the impact on a 
different outcome measure within the same outcome domain. Details on the differences between each of the 
findings reported here and those reported from prespecified analyses will be included in the data set to be released 
after this report. 


Educational Unofficial 

Intervention Comparison Outcome Domain Level WWC Evidence Rating 

Scale-up 

KIPP 

Option of No Option of English Language Unofficially Meets WWC Standards 

Continuing KIPP Continuing KIPP Arts High 2Gnoat with Reservations ae 

Option of No Option of English Language High School Unofficially Does Not Meet WWC _ 

Continuing KIPP Continuing KIPP Arts 9 Standards 

Reading Recovery 

Reading Recovery | Business as Usual English Language Elementary Unofficially Meets wwe Standards +1.167* 
Arts School without Reservations 

Reading Recovery | Business as Usual English Language Elementary Unofficially Meets wwe Standards +1.065* 
Arts School without Reservations 

Reading Recovery | Business as Usual English Language Elementary Unofficially Meets Wwe Standards 40.949" 
Arts School without Reservations 

Reading Recovery | Business as Usual English Language Elementary Unofficially Meets wwe Standards 40.641" 
Arts School without Reservations 

Reading Recovery | Business as Usual English Language Elementary Unofficially Meets Wwe Standards 40.589" 
Arts School without Reservations 

Reading Recovery | Business as Usual English Language Elementary Unofficially Meets wwe Standards 40.546" 
Arts School without Reservations 

: ‘ ici t 

Reading Recovery | Business as Usual English Language Elementary Unofficially Meets Wwe Standards 40.521" 
Arts School without Reservations 

Reading Recovery | Business as Usual English Language Elementary Unofficially Meets Wwe Standards +0.498* 
Arts School without Reservations 

Reading Recovery | Business as Usual English Language Elementary Unofficially Meets wwe Standards 40.474" 
Arts School without Reservations 

; ; ici t 

Reading Recovery | Business as Usual English Language Elementary Unofficially Meets wwe Standards 40.467" 
Arts School without Reservations 

Reading Recovery | Business as Usual English Language Elementary Unofficially Meets wwe Standards 40.444" 
Arts School without Reservations 
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Intervention 


Educational 
Level 


Unofficial 


Comparison 


Success for All (SFA) 


Outcome Domain | 


WWC Evidence Rating 


SEA Bicinees selena English Language Elementary Unofficially Meets wwe Standards 40.177 
Arts School without Reservations 
SEA Bhinceeal English Language Elementary Unofficially Meets wwe Standards +0.091 
Arts School without Reservations 
SEA Busineaeas Usual English Language Elementary Unofficially Meets wwe Standards +0.050 
Arts School without Reservations 
SEA Bucideeeas Usual English Language Elementary Unofficially Meets wwe Standards +0.050 
Arts School without Reservations 
SEA BusinessacUsual English Language Elementary Unofficially Meets wwe Standards +0.029 
Arts School without Reservations 
SEA picieenaeticial English Language Elementary Unofficially Meets wwe Standards +0.028 
Arts School without Reservations 
SEA Busines oaclenal English Language Elementary Unofficially Meets wwe Standards 40.013 
Arts School without Reservations 
SEA pasinasdaeitieual English Language Elementary Unofficially Meets wwe Standards 0.004 
Arts School without Reservations 
SEA Bucneseae USual English Language Elementary Unofficially Meets wwe Standards 0.019 
Arts School without Reservations 
SEA SusiessasUsual English Language Elementary Unofficially Meets wwe Standards 0.020 
Arts School without Reservations 
SEA Bhinceaeadl English Language Elementary Unofficially Meets wwe Standards 0.045 
Arts School without Reservations 
SEA Bicineseasiidl English Language Elementary Unofficially Meets wwe Standards 0.047 
Arts School without Reservations 
SEA Buciessas USual English Language Elementary Unofficially Meets wwe Standards 0.116 
Arts School without Reservations 
Teach for America (TFA) 
Early Childhood ae 
TEA Bicineeeaeiedal English Language and Elementary Unofficially Meets wwe Standards 40.138 
Arts without Reservations 
School 
Early Childhood = 
TEA Guiness aeUsual English Language and Elementary Unofficially Meets wwe Standards 40.131" 
Arts without Reservations 
School 
Early Childhood oe 
TEA Business agli English Language and Elementary Unofficially Meets wwe Standards 40.085 
Arts without Reservations 
School 
Early Childhood wd 
TEA Susinessaseial English Language and Elementary Unofficially Meets wwe Standards +0.030 
Arts without Reservations 
School 
TEA Busiveaeas Usual English Language Elementary Unofficially Meets wwe Standards 0.049 
Arts School without Reservations 
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Intervention 


Comparison 


Outcome Domain 


Educational 
Level 


Unofficial 


Early Childhood 


WWC Evidence Rating 


Unofficially Meets WWC Standards 


TFA Business as Usual Mathematics and Elementary ; . +0.142 
without Reservations 
School 
Early Childhood ea 
: ts WWC Standar 
TFA Business as Usual Mathematics and Elementary one ipa e slandale +0.095 
without Reservations 
School 
Early Childhood ~ 
TFA Business as Usual Mathematics and Elementary meme ied like Slat +0.060 
without Reservations 
School 
Early Childhood 7 
TFA Business as Usual Mathematics and Elementary eee aie: ee slanidalds +0.041 
without Reservations 
School 
TFA Business as Usual Mathematics eemenely Hnomeay heels ae Siena +0.009 
School without Reservations 


Validation 


Advanced ASSET Professional Development 


Elementary 


No Current 
Advanced ASSET | Advanced ASSET 
Professional Professional 7 
El t fficially Meets WWC Standards 
Development after | Development after Mathematics oe Huotic Jregaige +0.304 
f : ‘ : School with Reservations 
Science: It's Prior Experience 
Elementary with Science: It's 
Elementary 
No Current 
Advanced ASSET | Advanced ASSET 
Professional Professional = 
. nofficially Meets WWC Stan 
Development after | Development after Mathematics Eelienily one sa ia ska eats +0.204 
f ; ; . School with Reservations 
Science: It's Prior Experience 
Elementary with Science: It's 
Elementary 
No Current 
Advanced ASSET | Advanced ASSET 
plolessate piles) . Elementary | Unofficially Does Not Meet WWC 
Development after | Development after Mathematics - 
: ; : ; School Standards 
Science: It's Prior Experience 
Elementary with Science: It's 
Elementary 
No Current 
Advanced ASSET | Advanced ASSET 
Plclessione cilessione Elementary Unofficially Does Not Meet WWC 
Development after | Development after Mathematics = 
: ; : : School Standards 
Science: It's Prior Experience 
Elementary with Science: It's 
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Educational Unofficial 


Intervention Comparison Outcome Domain Level WWC Evidence Rating 
ssa aa No Current or Prior 
Development after State-wide Science hatheniages Elementary Unofficially Meets wwe Standards 40.658" 
‘ ; Curriculum or School with Reservations 
Science: It's es as 
Training Initiatives 
Elementary 
oe aaa No Current or Prior 
Development after State-wide Science piaineniates Elementary Unofficially Meets wwe Standards 40.489 
; ; Curriculum or School with Reservations 
Science: It's ey beg 
Training Initiatives 
Elementary 
lena aed No Current or Prior 
Development after State-wide Science einen Elementary Unofficially Meets wwe Standards 40.290 
: ine Curriculum or School with Reservations 
Science: It's a oe 
Training Initiatives 
Elementary 
sales aa No Current or Prior 
State-wide Science : Elementary Unofficially Meets WWC Standards 
Development after Mathematics f . -0.066 
. ; Curriculum or School with Reservations 
Science: It's se eee 
Training Initiatives 
Elementary 
Children’s Literacy Initiative (CLI) Program 
CLI pucnecaactienl English Language Elementary Unofficially Meets wwe Standards 40.198" 
Arts School without Reservations 
CLI Bisiese ae Ueual English Language Elementary Unofficially Meets wwe Standards 40.163* 
Arts School without Reservations 
CLI Busnesaae Usual English Language Elementary Unofficially Meets wwe Standards 40.114 
Arts School without Reservations 
College Readiness Program (CRP) 
CRPa SusiessasWeual General High School Unofficially Meets wwe Standards +1092" 
Achievement with Reservations 
5 — 
CRP pinceeasaial General High School Unofficially Meets wwe Standards +0.895* 
Achievement with Reservations 
= — 
CRP pusicseasuendl General High School Unofficially Meets wwe Standards +0.885* 
Achievement with Reservations 
; a 
CRP BagaascasUsual General High School Unofficially Meets wwe Standards 40.796* 
Achievement with Reservations 
- = 
CRP Busiieseacuaual General High School Unofficially Meets wwe Standards 40.779* 
Achievement with Reservations 
a ; ici ts WWC St 
CRP bicimaseac Sal General High School Unofficially Meets c Standards 40.760" 
Achievement with Reservations 
> aa 
CRP Baanessastigaal General High School Unofficially Meets wwe Standards +0.750* 
Achievement with Reservations 
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Educational Unofficial 
Intervention Comparison Outcome Domain Level WWC Evidence Rating 
Diplomas Now 
Educational . 3 
Diplomas Now Business as Usual Progress And Meee ea mnemee! y Meet bbe Slandals +0.062 
School without Reservations 
Attainment 
Educational ’ . er 
Diplomas Now Business as Usual Progress And nedeena Tian poemee! u Mes “we slanidals +0.050 
School without Reservations 
Attainment 
Educational : : - 
Diplomas Now Business as Usual Progress And Malena High ee ae MAN SlaMiats +0.018 
School without Reservations 
Attainment 
Educational - 
Diplomas Now Buciesdas Usual Progress And Middle and High Unofficially Meets wwe Standards 40.016 
School without Reservations 
Attainment 
GO College (An Enhanced Version of Talent Search) 
GO College Whole Busieseaslicial English Language High School Unofficially Meets wwe Standards 40.270 
School Program Arts with Reservations 
GO College Whole 
School Program GO College Whole English Language High School Unofficially Meets wwe Standards 40.054 
with Learning School Program Arts without Reservations 
Community 
GO College Whole Bishessaslaial haineniaties High School Unofficially Meets wwe Standards 40.070 
School Program with Reservations 
GO College Whole 
School Program GO College Whole hrainennanes High School Unofficially Meets wwe Standards 0.041 
with Learning School Program without Reservations 
Community 
Educational - 
BO poled: Whe Business as Usual progress and High School paomcialy eels bee Seat +0,533 
School Program with Reservations 
attainment 
GO College Whole Eanciteadl 
School Program GO College Whole Unofficially Meets WWC Standards 
: progress and High School : : -0.062 
with Learning School Program : without Reservations 
attainment 
Community 
GO College Whole Educational 
School Program GO College Whole Unofficially Meets WWC Standards 
‘ progress and High School : ; -0.004 
with Learning School Program with Reservations 
attainment 
Community 
GO College Whole Educational 
School Program GO College Whole ; Unofficially Meets WWC Standards 
; progress and High School ; ; -0.013 
with Learning School Program ; with Reservations 
; attainment 
Community 
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Intervention 


Outcome Domain 


Educational 
Level 


Unofficial 


Comparison 


Northeast Tennessee College and Career Ready Consortium 


WWC Evidence Rating 


Consortium® Business as Usual oe High School Hnoticaly Mee wwe slandalds -0.007 
Achievement with Reservations 
Educational a 
Consortium® Business as Usual progress and High School nnetiely eee lee Sinai +0.519* 
; with Reservations 
attainment 
Education Unofficially Meets WWC Standards 
Consortium® Business as Usual progress and High School y +0.081* 
. with Reservations 
attainment 
Esicatena Unofficially Meets WWC Standards 
Consortium® Business as Usual progress and High School y +0.054 
. with Reservations 
attainment 
Consortium® Business as Usual cared High School | ces vie slanted: +1.375* 
Achievement with Reservations 
Consortium® Business as Usual Seer High School Unotneally eet WWE olgnalds +0.341 
Achievement with Reservations 
Consortium> Business as Usual clan High School Hhomcely ee mw Slane +0.004 
Achievement with Reservations 
Consortium® Business as Usual eeliele! High School Unetnetally Mees mwe Siena -0.018 
Achievement with Reservations 
Consortium® Business as Usual een High School snoteealy ue wwe Standards -0.620 
Achievement with Reservations 
Educational + 
; ; lly Meets WWC St 
Consortium® Business as Usual progress and High School phones y — © elandals +0.529* 
: with Reservations 
attainment 
Ealcalonl Unofficially Meets WWC Standards 
Consortium® Business as Usual progress and High School y +0.010 
: with Reservations 
attainment 
Reading Apprenticeship (RA) 
Reading . Paiessasieudl English Language High School Unofficially Meets wwe Standards 40.327* 
Apprenticeship Arts without Reservations 
Reading Enhances Achievement During the Summer 
(READS) 
READS Businesses English Language Elementary Unofficially Meets wwe Standards 40.056 
Arts School without Reservations 
READS pisiiessasWeual English Language Elementary Unofficially Meets wwe Standards 40.045" 
Arts School without Reservations 
READS Bidiqaesaaslal English Language Elementary Unofficially Meets wwe Standards 40.034" 
Arts School without Reservations 
READS Bisinees aaa English Language Elementary Unofficially Meets wwe Standards +0.020 
Arts School without Reservations 
READS Bucieeeae Usual English Language Elementary Unofficially Meets wwe Standards 0.007 
Arts School without Reservations 
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Educational 


Unofficial 


Intervention Comparison Outcome Domain Level WWC Evidence Rating 
READS BusinessasUSual English Language Elementary Unofficially Meets wwe Standards 0.053 
Arts School without Reservations 
READS - PIF BusineseasUaual English Language Elementary Unofficially Meets wwe Standards 41.157 
Arts School without Reservations 
READS - PIF Bicincee seal English Language Elementary Unofficially Meets wwe Standards 40.873* 
Arts School without Reservations 
StartSmart K-3 Plus Program 
Extended School Bucinessas Usual English Language Elementary Unofficially Meets Wwe Standards 40.310" 
Year Arts School without Reservations 
Extended School BusiieseacUsial English Language Elementary Unofficially Meets Wwe Standards 40.278 
Year Arts School without Reservations 
Extended School Bucinesoasiieial English Language Elementary Unofficially Meets Wwe Standards 40.212" 
Year Arts School without Reservations 
Extended School paanecsaeiicial English Language Elementary Unofficially Meets wwe Standards 40.194* 
Year Arts School without Reservations 
Extended School Bisineseas Usual English Language Elementary Unofficially Meets Wwe Standards +0.099* 
Year Arts School without Reservations 
Extended School Pusinecsastlcudl English Language Elementary Unofficially Meets wwe Standards 40.059 
Year Arts School without Reservations 
Extended School Busiiessasusial English Language Elementary Unofficially Meets Wwe Standards 40.046 
Year Arts School without Reservations 
Extended School Bicincseaalisial English Language Elementary Unofficially Meets wwe Standards 40.041 
Year Arts School without Reservations 
Extended School Becieseasuel seainennaties Elementary Unofficially Meets wwe Standards 40.202 
Year School without Reservations 
Extended School BusiaessasUsial iiatnemiaties Elementary Unofficially Meets wwe Standards 40.156* 
Year School without Reservations 
Virginia Initiative for Science Teaching and Achievement (VISTA) 
Hein Team! ; i Elementary Unofficially Meets WWC Standards 
Professional Business as Usual Science ; ; +0.068 
School with Reservations 
Development 
VISTA Teacher a 
Professional Business as Usual Science Middle School ome Does Nopuect we 7 
Standards 
Development 
a Teach ; Unofficially Does Not Meet WWC 
Professional Business as Usual Science High School - 
Standards 
Development 
Development 
Advanced Placement (AP) Insight 
‘ . F : ici t 
AP Insight Business as Usual Science High School saonciely phi ve sleuaie> +0.199 
with Reservations 
AP Insight Business as Usual Science High School Hnomeany Nee lee See +0.128 
with Reservations 
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Intervention Comparison 


AP Insight Business as Usual 


Outcome Domain 


Science 


AP Insight Business as Usual 


AVID schoolwide 


Business as Usual 


Science 


Advancement via Individual Determination (AVID) 


English Language 


intervention Arts 
uIP Senne Business as Usual Mathematics 
intervention 
AVID schoolwide . Euan 
; Business as Usual progress and 
intervention ; 
attainment 
Around the Corner (ATC) 


Educational 
Level 


High School 


High School 


Middle and High 
School 
Middle and High 
School 


High School 


Early Childhood 


Unofficial 
WWC Evidence Rating 


Unofficially Meets WWC Standards 
with Reservations 
Unofficially Meets WWC Standards 
with Reservations 


Unofficially Does Not Meet WWC 
Standards 
Unofficially Does Not Meet WWC 
Standards 


Unofficially Does Not Meet WWC 
Standards 


Agoundthe Gamer Existing SFA English Language and Elementary Unofficially Meets wwe Standards 
Program Arts without Reservations 
School 
Existing SFA English Language eal ean Unofficially Meets WWC Standards 
Around the Corner and Elementary ‘ f 
Program Arts without Reservations 
School 
Existing SFA English Language ally Cudiope Unofficially Meets WWC Standards 
Around the Corner and Elementary 
Program Arts without Reservations 
School 
Existing SFA English Language aly Sncnoee Unofficially Meets WWC Standards 
Around the Corner and Elementary 
Program Arts without Reservations 
School 
Existing SFA English Language aly Guucnooe Unofficially Meets WWC Standards 
Around the Corner and Elementary ; 
Program Arts without Reservations 
School 
Existing SFA English Language Bally Caution Unofficially Meets WWC Standards 
Around the Corner and Elementary 
Program Arts without Reservations 
School 
Existing SFA English Language Ea uudnone Unofficially Meets WWC Standards 
Around the Corner and Elementary 
Program Arts without Reservations 
School 
Existing SFA English Language Ea wees Unofficially Meets WWC Standards 
Around the Corner and Elementary 
Program Arts with Reservations 
School 
gs Early Childhood am 
Aiounaine Conic: Existing SFA English Language and Elementary Unofficially Meets wwe Standards +0.007 
Program Arts with Reservations 
School 
Arts Achieve: Impacting Student Success in the Arts 
Elementary, of 
; ,; ; fficially Does Not Meet WW 
Arts Achieve* Business as Usual Engh Sariguege Middle, and High SUNY Dees NURMES C - 
Arts Schou Standards 
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Intervention 


Outcome Domain 


Educational 
Level 


Unofficial 


Comparison 


Arts for Learning (A4L) Lessons 


WWC Evidence Rating 


AL Business ceed English Language Elementary Unofficially Meets wwe Standards 0.009 
Arts School without Reservations 
MAL A nceieeal English Language Elementary Unofficially Meets wwe Standards 0.029 
Arts School without Reservations 
Bay State Reading Institute (BSRI) 
Bay State Reading Aucinescaelicual English Language Elementary Unofficially Does Not Meet WWC _ 
Institute Arts School Standards 
Bay State Reading Bucinessasdlaual English Language Elementary Unofficially Does Not Meet WWC _ 
Institute Arts School Standards 
Bay State Reading pasieseasGual English Language Elementary Unofficially Does Not Meet WWC _ 
Institute Arts School Standards 
Bay State Reading Biisineseaellsuial English Language Elementary Unofficially Does Not Meet WWC _ 
Institute Arts School Standards 
Bay State Reading Busessas leual English Language Elementary Unofficially Does Not Meet WWC _ 
Institute Arts School Standards 
Boston Teacher Residency (BTR) 
Newly Hired BTR English Language Elementary and | Unofficially Meets WWC Standards : 
Graduates Bisiiess asus Arts Middle School with Reservations ree 
Newly Hired BTR Bugnessas Usual iiathaniaties Elementary and Unofficially Meets wwe Standards 0.054 
Graduates Middle School with Reservations 
Building Assets, Reducing Risks (BARR) Model 
BARR sucnessaslieual English Language High School Unofficially Meets wwe Standards 40.238" 
Arts without Reservations 
BARR Bisiessacitieual English Language High School Unofficially Meets wwe Standards 40.150" 
Arts without Reservations 
BARR Buciseae Ueual English Language High School Unofficially Meets wwe Standards 40.139 
Arts without Reservations 
BARR Business as Usual Mathematics High School pao pees ia Stiendaids +0.518* 
without Reservations 
BARR Business as Usual Mathematics High School Peete by Meets wile Sianeli +0.381* 
without Reservations 
BARR Business as Usual Mathematics High School | lig ee haan +0.333* 
without Reservations 
Educational 7 
BARR Business as Usual Progress And High School mney —_ biti Suneens +0.251* 
‘ without Reservations 
Attainment 
Educational ss 
BARR Business as Usual Progress And High School Hnemealy nie ule Slandalus +0,194 
aaeinnner without Reservations 
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Intervention 


Comparison 


Outcome Domain 


Educational 
Level 


Unofficial 
WWC Evidence Rating 


Collaborative Organizational Model to Promote Aligned Support Structures (COMPASS) 


COMPASS eicnessaeluaial English Language Elementary Unofficially Does Not Meet WWC 
Arts School Standards 
COMPASS Fiisqessaelisial English Language Elementary Unofficially Does Not Meet WWC 
Arts School Standards 
COMPASS Bucsnessae sual English Language Elementary Unofficially Does Not Meet WWC 
Arts School Standards 
COMPASS Basiessaa Usual English Language Middle School Unofficially Does Not Meet WWC 
Arts Standards 
COMPASS Business as Usual Eee Middle School ppemcjelly Ubes Natick wie 
Arts Standards 
COMPASS Business as Usual Engen Eanguede Middle School pnomea ny Does Nat Meet WN 
Arts Standards 
CollegeYES 
CollegeYES Business as Usual Science High School Unofficially a Wwe 
CollegeYES Business as Usual Science High School Unofficially pecans Wwe 
CollegeYES Business as Usual Science High School Unofficially a Wwe 
CollegeYES Business as Usual Science High School Unofficially ae an Meet WWC 
CollegeYES Business as Usual Science High School Unoincilly alia wwe 
Academic ia 
CollegeYES Business as Usual Performance In High School Unotficially pn ae 
School 
Academic i 
CollegeYES Business as Usual Performance In High School Dnaingely a WANG 
School 
Academic a 
CollegeYES Business as Usual Performance In High School aie oa was 
School 
Educational Ss 
CollegeYES Business as Usual Progress And High School Unofficially a WwcCc 
Attainment 
Educational he 
CollegeYES Business as Usual Progress And High School UnaiiGelly ae eae WONG 
Attainment 
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Unofficial 


Intervention Outcome Domain Level WWC Evidence Rating 


| Educational 


Comparison 


Convergence Academies (CA) 


; English Language Elementary and Unofficially Ineligible for WWC _ 

on eee eme Arts Middle School Review 
ici igible for WW 

CA panes ae Gua niaihennaties Elementary and Unofficially Ineligi e for WWC _ 

Middle School Review 
Curriculum 2.0 

Cunieulin2.0 Suciiasead Ula English Language Elementary Unofficially Ineligible for WWC _ 
Arts School Review 

Giiiautunex0 Buciesdeae Usual English Language Elementary Unofficially Ineligible for WWC _ 
Arts School Review 

Cunteitun 2.0 Buisassasiieual English Language Elementary Unofficially Ineligible for WWC _ 
Arts School Review 


Exceptional Coaching for Early Language and Literacy - Enhanced (ExCELL-E) 


English Language 


Unofficially Does Not Meet WWC 


EXCELL-E Business as Usual ie Early Childhood Siandards - 
; icially Meets WWC St 
EXCELL-E Business soleil English Language Elementary Unofficia y Meets c Standards 40.051 
Arts School with Reservations 
Facilitating Long-Term Improvements In Graduation and Higher Education for Tomorrow (FLIGHT) 
Educational > 
FLIGHT Business as Usual Progress And High School nomen odes NerMee ils - 
Standards 
Attainment 
Florida Master Teacher Initiative (FMTI) 
EMTI¢ Bisiess Asis English Language Elementary Unofficially Meets wwe Standards 0.005 
Arts School without Reservations 
EMTI¢ puciieseas cual English Language Elementary Unofficially Meets wwe Standards 0.022 
Arts School without Reservations 
EMTI¢ Bisnessas lisa English Language Elementary Unofficially Meets Wwe Standards 0.081 
Arts School with Reservations 
EMTI¢ Aucnescanusual English Language Elementary Unofficially Meets wwe Standards 0114 
Arts School with Reservations 
FMTI4 Business as Usual Mathematics eemeniay epoca sei we Slalioles +0.012 
School without Reservations 
FMTI¢ Business as Usual Mathematics Been phoney Meee we Slanidalis -0.024 
School without Reservations 
FMTI¢ Business as Usual Mathematics Bemenery enomealy ices bibs elandads +0.021 
School with Reservations 
FMTI4 Business as Usual Mathematics Eemenialy vpomcaly eet me landalds -0.080 
School with Reservations 
InnovateNYC Ecosystem 
Math Appse Business as Usual Mathematics Middle School | Mee MiG stiolde +0.246 
with Reservations 
fficially Meets WWC Standard 
Math Apps Business as Usual Mathematics Middle School moomiclally eet yn aianaatas +0.099 
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Educational 


Unofficial 


Intervention Comparison Outcome Domain Level WWC Evidence Rating 
Math Appse Business as Usual Mathematics Middle School voenicely Meets ie sleMalts -0.154 
with Reservations 
Math Apps® Business as Usual Mathematics Middle School ee eee - 
Standards 
Oakland Accelerates 
Oakland Busineseas Usual English Language High School Unofficially Ineligible for WWC _ 
Accelerates Arts Review 
Oakland Busincseas Usual English Language High School Unofficially Ineligible for WWC _ 
Accelerates Arts Review 
kl i ici igi 
Oakland Busiesé-as Usual English Language High School Unofficially Ineligible for WWC _ 
Accelerates Arts Review 
Gavel Business as Usual Mathematics High School shea nee ea - 
Accelerates Review 
eae Business as Usual Mathematics High School pnomeety hela ae - 
Accelerates Review 
eanane Business as Usual Mathematics High School Hnenncrally inekotle fee - 
Accelerates Review 
Oakland | Eee Unofficially Ineligible for WWC 
Business as Usual progress and High School ‘ - 
Accelerates Review 
attainment 
Oakland | Eaueavole Unofficially Ineligible for WWC 
Business as Usual progress and High School . - 
Accelerates Review 
attainment 
Oakland . Egesone . Unofficially Ineligible for WWC 
Business as Usual progress and High School ; - 
Accelerates Review 
attainment 
Oakland eaeanene! Unofficially Ineligible for WWC 
Business as Usual progress and High School . - 
Accelerates ; Review 
attainment 
Oakland . Eleaene Unofficially Ineligible for WWC 
Business as Usual progress and High School - 
Accelerates Review 
attainment 
Oakland erent Unofficially Ineligible for WWC 
Business as Usual progress and High School - 
Accelerates ; Review 
attainment 
Educational - ses 
Oakland ; Unofficially Ineligible for WWC 
Business as Usual progress and High School ; - 
Accelerates ; Review 
attainment 
Educational a x 
Oakland Unofficially Ineligible for WWC 
Business as Usual progress and High School - 
Accelerates ' Review 
attainment 
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F-12 


Intervention 


Outcome Domain 


Educational 
Level 


Unofficial 


Comparison 


Pathways to STEM Initiative (PSI) 


WWC Evidence Rating 


Unofficially Meets WWC Standards 


PSII Business as Usual Science Middle School ; +0.031 
with Reservations 
Problem-Based Learning (PBL) 
PBL Biicinescsaliual English Language High School Unofficially Ineligible for WWC _ 
Arts Review 
PBL Business as Usual Mathematics High School viomeey neve ore - 
Review 
PBL Business as Usual Science High School mete’ nel oe - 
Review 
' icially Ineligible for WW 
PBL Business as Usual Social Studies High School a cig igib Seg - 
Review 
PBL Biicincaeaa lua) General High School Unofficially Ineligible for WWC _ 
Achievement Review 
Rural Math Excellence Partnership (RMEP) 
ici it Meet WW 
RMEP Project Business as Usual Mathematics Medeandigh SPO Pee Re v 7 
School Standards 
ici Meet WW! 
RMEP Project Business as Usual Mathematics meeeand tig pnelnclelly Pees WOrMee : 7 
School Standards 
RMEP Project Business as Usual Mathematics Meds eae enemeay Meets as tanaas -0.190 
School with Reservations 
STEM Learning Opportunities Providing Equity (SLOPE) 
fficially Does Not Meet WW 
SLOPES Business as Usual Mathematics Middle School een ee : - 
Standards 
STEM Summer Learning with VEX Robotics 
STEM Summer 
Learning Program | No Summer hiathematies Middle Schoo! Unofficially Meets wwe Standards 40.181 
with VEX Learning Program with Reservations 
Robotics* 
STEM Summer 
Learning Program | No Summer picinenaties Middle School Unofficially Meets Wwe Standards 40.159 
with VEX Learning Program with Reservations 
Robotics* 
STEM Summer 
Learning Program | No summer Katheraties Middle School Unofficially Meets wwe Standards 40.100 
with VEX Learning Program with Reservations 
Robotics* 
STEM Summer 
Learning Program | No summer piarhcinanes Middle School Unofficially Meets wwe Standards 40.071 
with VEX Learning Program with Reservations 
Robotics? 
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Educational Unofficial 
Intervention Comparison Outcome Domain Level WWC Evidence Rating 
STEM Summer 
Learning Program | No Summer hathemaiies Middle School Unofficially Meets wwe Standards 0.090 
with VEX Learning Program with Reservations 
Robotics* 
Texas Tech University “Tech Teach” Program 
Competency : . ae 
Based Training Math Forward only Mathematics Middle and igh Unofneily dic mee plandalts +0.192* 
School with Reservations 
and Math Forward 
Competency : : 2 
Based Training Math Forward only Mathematics Medle andinigh Hnomnealy Des NotHeehwyie - 
School Standards 
and Math Forward 
Turnaround with Increased Learning Time (TILT) 
TILT! Bucincawae cial English Language Middle School Unofficially Meets wwe Standards 40.233* 
Arts with Reservations 
TILT! Business as Usual English Language Middle School unomeay eee bake lendalts +0.055 
Arts with Reservations 
TILT! Business as Usual Mathematics Middle School enomea)y disoded si Saidals +0.067* 
with Reservations 
TILT! Business as Usual Mathematics Middle School pone — oe Sinai -0.077* 
with Reservations 
TILT! Business as Usual Science Middle School ere Li alia elandaide -0.093 
with Reservations 
TILT! Business as Usual Science Middle School voomeay cere _ elanalds -0.105 
with Reservations 
We Are A Village 
We Are A Village Aisiaseaetisual English Language Elementary Unofficially Ineligible for WWC _ 
Arts School Review 
Write Up! 
Write Up! Bucnassaslicudl English Language Elementary Unofficially Meets wwe Standards 0111 
Arts School with Reservations 
Write Up! Biceee ascii English Language Elementary Unofficially Meets wwe Standards 0211 
Arts School with Reservations 
Write Up Sucneseuslisual English Language Middle and High Unofficially Meets Wwe Standards 0.075 
Arts School with Reservations 
Notes: 


* College Readiness Program was not implemented with adequate fidelity. 
» Northeast Tennessee College and Career Ready Consortium was not implemented with adequate fidelity. 
° Arts Achieve was not implemented with adequate fidelity. 
FMTI was not implemented with adequate fidelity. 

° Math Apps was not implemented with adequate fidelity. 
‘Pathways to STEM Initiative was not implemented with adequate fidelity. 
3 SLOPE was not implemented with adequate fidelity. 
"STEM Summer Learning Program with VEX Robotics was not implemented with adequate fidelity 
‘Turnaround with Increased Learning Time was not implemented with adequate fidelity. 

‘Write Up! was not implemented with adequate fidelity. 
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Appendix G: Alternative Approaches to Assessing 


Representativeness of the 67 i3 Impact Evaluations 


Chapter 4 reports the percentage of i3 impact evaluations that were based on a sample that was 
representative of the population served by the i3 intervention. An evaluation sample was not considered 
representative of this population if it excluded more than 25 percent of the schools that received the i3 
intervention (see Section 4.1.4 for more details). However, because thresholds of this type are inherently 
arbitrary, and the 25 percent threshold could be considered generous, the Abt Team explored whether the 
share of evaluations that satisfy the representativeness criteria would be much smaller under a more 
stringent threshold of 10 percent. 


Exhibit G.1 compares the results from the two approaches. It shows that changing this threshold from 25 
percent to 10 percent would reduce the percentage of i3 evaluations that are based on a representative 
sample from 79 percent to 63 percent. 


Exhibit G.1: Representativeness Results under Different Exclusion Thresholds for 
Schools that Received the i3 Intervention 
Representativeness of i3 Primary Approach - Cannot Exclude Alternative Approach - Cannot Exclude 


Impact Evaluations More than 25% of Schools More than 10% of Schools 


No prespecified analyses 1 
(1%) (1%) 
Yes 53 42 
(79%) (63%) 
No 13 24 
(19%) (36%) 


Note: Columns do not sum to 100% due to rounding. 
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Appendix H: Short-term and Long-term Goals Met by 67 i3 Impact and 
Implementation Evaluations 


xo} wn = 5 
2 x) | < 2g n 
s o © S = = = 
c Es tS o£ °'H = oO 
2 FS} = at = Lz ORs 
= 8 ao _| & a aa = 
= 5 2 S23) 5 2Be |s85 w 
= = ® | || = = mm ee) LF 
o cis = 32 2 a SoD = 2) 
a = ¢ — fe) 2S s+ SH 2 
=e cS @ He ors cD oc! FT + 5 os ro} 
—£s| a8 sp |oa8S| ua 252 25 6 = 
2Bl| ss ©£s |g S8| o ses 523 ” 
ae | ee [as |e ae er See |g | 5 
Intervention os Er 55 ESZsa\ = “aes ost = 
Scale-up 
KIPP v v v x No v v Yes 
Reading Recovery v v v v Yes v v Yes 
Success for All (SFA) v v v x No v x No 
Teach for America (TFA) v v v x No v x No 
Validation 
Advanced ASSET Professional gi V V y Yes i - No 
Development 
The Baby Family and Child y V m v No y ‘ No 
Education (FACE) Program 
Children’s Literacy Initiative V V v v Yes v v Yes 
(CLI) Program 
llaborati rategic Readin 
nies ative Strategic Reading vi Wy zy y; Yes y x No 
ne Readiness Program V Vv v v Yes x v No 
pach atlas Writers Program V yi Lp vee V eo Yes 
Diplomas Now v v v Yes v x No 
enhancing Missouri's 
Instructional Networked v v v v Yes v v Yes 
Teaching Strategies (eMINTS) 
GO College (An Enhanced 7 V Vv v Yes v x No 
Version of Talent Search) 
Leadership Assistance for 
Science Education Reform v v v x No v x No 
(LASER) 
Northeast Tennessee College V Yes x v No 
and Career Ready Consortium 
Reading Apprenticeship (RA) Yes x No 
Reading Enhances Achievement ¥ Yes " No 
During the Summer (READS) 
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3 aa |e 5 aie | =! 
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2 5 Es = = o£ 5 o 
2 |8 |S (|se /|z 2% Gea | 3 
5 S fs) eos lee 23s $55 Wi 
= = ® Seaele eee i cel J 
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>8/ 3S Ss |x SBl un ees eS n 
| ef | £8 |e S3/5 Ss SEs ir 
SS| 23 |2s8 |2£35/ w Sse )/s82 uw 
Intervention oO o Er a’) £3 sas) = us & io = 
StartSmart K-3 Plus Program v v v v Yes v v Yes 
Virginia Initiative for Science 
Teaching and Achievement v v v v Yes v x No 
(VISTA) 
Develon 
ae ie ievement Network uy Gi v v Yes x x No 
e 
fal Placement (AP) ov 74 v v Yes v x No 
Advancement via Individual - y i v No v 2 No 
Determination (AVID) 
Around the Corner (ATC) v v v v Yes v x No 
Arts Achieve: Impacting Student v e, e, V Yes : me No 
Success in the Arts 
Arts for Learning (A4L) Lessons v v v v Yes x No 
ee ae Reading Institute e v va v Yes x No 
ane Teacher Residency Y v v v Yes v x No 
Building Assets, Reducing Risks V e V . No V ey Yes 
(BARR) Model 
Collaborative Organizational 
Model to Promote Aligned v v x x No v x No 
Support Structures (COMPASS) 
CollegeYES v of x No v No 
Convergence Academies v v v No v No 
Curriculum 2.0 x v v No x No 
Data-Driven Decision Making 
and Information Technology v v v v Yes v x No 
Curricula in Schools 
E3TL Teacher Performance V uy . vy No y, es No 
Evaluation System 
EngageMe P.L.E.A.S.E. v v Yes v No 
Every Child Ready (ECR) v x x v No v No 
Everyday Arts for Special 
; v v v v Yes v x No 
Education (EASE) 
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Intervention 


Quality implementation 


evaluation 


Impact evaluation considered 


independent 


Unofficially Meets WWC 


Standards 


Impact evaluation sample 


adequately representative of 


population served 


MEETS SHORT-TERM GOALS 
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Evidence that the intervention 
was implemented with 


adequate fidelity 
impact on student outcomes 


MEETS LONG-TERM GOALS 


@ 
< 
) 
~ 
wn 
GS 
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~ 
Oi 
— 
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@ 
=) 
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® 
= 
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Exceptional Coaching for Early 

Language and Literacy - v v v v Yes v x No 
Enhanced (ExCELL-E) 

The Expository Reading and Y V V : No V Vv Jax 
Writing Course (ERWC) 

Facilitating Long-Term 

Improvements in Graduation and ¥ ue V y Yes v re No 
Higher Education for Tomorrow 

(FLIGHT) 

eat Master Teacher Initiative 2 Yes : , No 
InnovateNYC Ecosystem va Yes x x No 
ie vy a 
Integrating English Language UY ey em V No av 2 No 
Development (ELD) and Science 

Internet-based Reading 

Apprenticeship Improving v v x v No v x No 
Science Education (iIRAISE) 

Making Time for What Matters y, y e y No . y No 
Most 

New England Network for 

Personalization and v v v v Yes v x No 
Performance (NETWORK) 

Oakland Accelerates v v x v No v x No 
Ounce of Prevention Fund 

Professional Development v v v v Yes v x No 
Initiative (PDI) 

Pathways to STEM Initiative ue v v Yes x x No 
(PSI) 

Problem-Based Learning (PBL)@ v v x No v No 
PTA Comunitario v v x No v No 
Public School Choice Initiative 

(PSC) v v x x No v x No 
Responding Effectively to 

Assessments with Curriculum v v x x No v x No 
and Teaching (REACT) 
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Evidence that the intervention 
was implemented with 


MEETS SHORT-TERM GOALS 
adequate fidelity 


Impact evaluation considered 
independent 

adequately representative of 
impact on student outcomes 
MEETS LONG-TERM GOALS 


Unofficially Meets WWC 
population served 


Standards 
Impact evaluation sample 
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Quality implementation 


evaluation 


Intervention 


Rio Grande Valley Center for 
Teaching and Leading v v v v Yes v x No 
Excellence: New Teacher 
Training (NTT) 
Rural Math Excellence : V V y No e No 
Partnership (RMEP) 
School of One v v v v Yes v x No 
Schools to Watch (STW): School 

v v v v v x 
Transformation Network Project nes Ng 
Science, Technology, Education 
and Math Education for the 21st v v v v Yes v x No 
Century (STEM21) 
Spheres of Proud Achievement 
in Reading for Kids (SPARK) v v v v Yes v v Yes 
Program 
STEM Learning Opportunities V V V y Yes ; rn No 
Providing Equity (SLOPE) 
STEM Summer Learning with 

v v v v x x 
VEX Robotics Yee a2 
Texas Tech University “Tech V V ‘ V No V ¥e No 
Teach” Program 
Transforming Teacher Talent y v ie Wy No = i No 
(t8) System 
Turnaround with Increased 

v v v v v 
Learning Time (TILT) ue e Ne 
We Are A Village v v x x No v x No 
Write Up! v v v v Yes x x No 
Total number of evaluations 
that meet the short- and long- 40 9 
term goals of i3 (60%) (13%) 
(N=67 evaluations) 


Notes: 

¥ indicates that the evaluation met the expectation; 

x indicates that the evaluation did not meet the expectation; 
* The impact evaluation of PBL did not prespecify analyses. 
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Appendix |: Alternative Approaches to Summarizing the Findings from 


the 67 i3 Impact Evaluations 


Chapter 4 reports the percentage of i3 interventions that improved student academic outcomes, by 
summarizing findings from prespecified analyses that Unofficially Meet WWC Standards with or without 
Reservations. An intervention was considered to have improved student academic outcomes if it had a 
positive and statistically significant impact on at least one student academic outcome and no negative and 
statistically significant impacts on these outcomes, after correcting for multiple comparisons.” This 
appendix explores whether the percentage of interventions that improved student academic outcomes is 
sensitive to (1) the method of summarizing multiple findings or (2) the inclusion of findings based on 
analyses that were not prespecified.” 


Alternative Methods for Summarizing Multiple Findings 


We explored two alternative approaches to summarizing multiple findings. The first alternative used the 
most common, or modal, finding reported by the evaluation. Each finding was classified into one of the 
following categories: 


e Positive if the estimate was positive and statistically significant, without any multiple comparisons 
adjustment; 


e Negative if the estimate was negative and statistically significant, without any multiple comparisons 
adjustment; or 


e Null if the estimate was not statistically significant. 


Under this alternative, the summary finding for the intervention is based on the modal category across 
findings. An intervention was considered to have improved student academic outcomes if the modal 
category of findings was positive. 


The second alternative used the average effect size across the impact estimates. We computed effect sizes 
from the estimates reported for prespecified analyses using procedures from the WWC Procedures & 
Standards Handbook, Version 3.0, calculated the simple average of those effect sizes, and tested whether 
the average effect size was statistically significant. It is important to note that this hypothesis test is 
affected by the correlation among the impact estimates caused by the fact that they were based on the 
same or similar samples. For each evaluation, we either determined that significance of the average effect 
size was not dependent on the size of the correlation—that is, the conclusion would be the same for all 
plausible values of the correlation—or we were able to estimate the correlation using information reported 
by the evaluator. 


Under the second alternative, an intervention was considered to have improved student academic 
outcomes if the average effect size is positive and statistically significant—and not to have improved 
student academic outcomes if (1) the average effect size was negative and statistically significant or (2) 
the average effect size was not statistically significant (null). 


© For a description of how multiple comparisons adjustments were made, see Appendix B. 


”| Findings reported by evaluators from analyses that were not prespecified can be found in Appendix F. 
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Exhibit I.1 compares the results from the two alternative approaches to the results from our primary 
approach reported in Chapter 4. Exhibits I.2 and I.3 show the pie charts that correspond to the alternative 
approaches, and that can be compared to Exhibit 4.6 in the report. Both alternatives give very similar 
results to our primary approach. ” 


Exhibit 1.1: Summary of the Impacts on Student Academic Outcomes Using Our 
Primary Approach and Two Alternative Approaches (N=67 Impact 
Evaluations) 


Alternative #1 - Alternative #2 - 
Primary Approach Modal Finding Average Effect Size 


Positive 


12 10 11 


” The findings that are summarized in these three analyses can be found in Appendix D. 
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Exhibit I.2: | Summary of Whether the i3 Impact Evaluations Found Statistically 
Significant Positive Impacts on Student Academic Outcomes — Based on 
Modal Findings 
All Grant Types mYes 
(67 Evaluations) No (null) 
1 @ No (negative) 
8 (2%) 10 © No Evidence (Unofficially Does Not Meet WWC Standards) 
E> (12%) (15%) ®@ No Evidence (Unofficially Ineligible for WWC Review) 


® No Evidence (no prespecified analyses) 


(56%) 


Scale-up Validation Development 
(4 Evaluations) (15 Evaluations) (48 Evaluations) 


5 
(33%) 
2 


(50%) 
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Exhibit 1.3: Summary of Whether the i3 Impact Evaluations Found Statistically 
Significant Positive Impacts on Student Academic Outcomes — Based on 
Average Effect Size 
All Grant Types myYes 
(67 Evaluations) @ No (null) 
@ No (negative) 
8 ( Fe ) 11 @ No Evidence (Unofficially Does Not Meet WWC Standards) 
(12%) (16%) ®@ No Evidence (Unofficially Ineligible for WWC Review) 


@ No Evidence (no prespecified analyses) 


35 
(52%) 


Scale-up Validation Development 
(4 Evaluations) (15 Evaluations) (48 Evaluations) 


2 2 


(50%) (50%) 
(50%) 


Inclusion of Non-Prespecified Analyses 


This report focuses primarily on findings from analyses that were prespecified by the evaluators (see 
Chapter 3). However, we collected all of the findings that the evaluators were willing to share, whether or 
not they were from prespecified analyses.”’ We conducted a sensitivity analysis to assess whether the 
conclusions about the impacts of the i3 interventions on student academic outcomes presented in Chapter 
4 would be substantially different if the assessment had included findings from both prespecified and non- 
prespecified analyses. 


In principle, including findings from non-prespecified analyses could either increase or decrease the 
percentage of i3 interventions that found at least one statistically significant positive impact on student 
academic outcomes. If evaluators selectively reported more favorable findings from non-prespecified 
analysis, we would expect the inclusion of these findings to increase the percentage of interventions that 


’3 Findings reported by evaluators from analyses that were not prespecified can be found in Appendix F. 
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had a significant positive impact on at least one student academic outcome. However, if the analyses that 
were not prespecified are ones that would increase the number of comparisons among which we will 
apply multiple comparisons corrections, we might expect the inclusion of these findings to decrease the 
percentage of interventions that had a significant positive impact on at least one student academic 
outcome. 


Exhibit I.4 shows the distribution of summary findings, based on findings from both prespecified and 
non-prespecified analyses, using our primary approach to summarizing multiple findings. Including 
findings from non-prespecified analyses changes the summary impact finding for only six evaluations. 
The evaluation that did not prespecify analyses receives a summary impact finding of “no evidence” 
because all of the findings from non-prespecified analyses were Unofficially Ineligible for WWC Review. 
Including findings from non-prespecified analyses reduces the number of evaluations with null findings 
by five; two of these evaluations would be summarized as having negative impact findings, one as having 
mixed findings, and one as having positive findings. The remaining evaluation receives a summary 
impact finding of “no evidence’ because including findings from non-prespecified analyses increases the 
number of findings that Unofficially DNM WWC Standards such that it is now the modal rating for that 
evaluation. Including these additional findings in our summary does not change the conclusion that 
almost one-fifth of the i3 interventions improved student academic outcomes. 
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Exhibit 1.4: | Summary of Whether the i3 Impact Evaluations Found Statistically 
Significant Positive Impacts on Student Academic Outcomes - Including 
Prespecified and Non-Prespecified Analyses 
All Grant Types a 
(67 Evaluations) @ No (null) 
® No (mixed) 
9 13 @ No (negative) 
(13%) ( 19%) ™ No Evidence (Unofficially Does Not Meet WWC Standards) 


® No Evidence (Unofficially Ineligible for WWC Review) 


Scale-up Validation Development 
(4 Evaluations) (15 Evaluations) (48 Evaluations) 


8 
(53%) 


(48%) 
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